An Introduction to Evaluation. Chris Fox
never consciously or at least structurally articulated.
Other techniques like focus group discussions or stakeholder meetings can also be used. Ultimately however the evaluator will act as the TOC proponent based on what they piece together. The last step is to procure validation and finally generate agreement around what the TOC of the programme could have been, had it been conceptualised at the planning stage. At this point, and in many cases, agreement is not sought to allow the project to be implemented – it is sought to ensure that the evaluation findings will not be contested on the basis of its inexistence.
Case Study ‘Project Superwoman’ theory of change
This is an example of a theory of change exercise.
The Center for Theory of Change is a non-profit organisation established to promote quality standards and best practice for the development and implementation of Theory of Change. They have placed a number of examples of theory of change exercises on their website. This theory of change exercise for a programme called ‘Project Superwoman’ was created by ActKnowledge in partnership with the Aspen Institute Roundtable on Community Change.
Project Superwomen began as a collaboration between a social service provider, a non-profit employment training centre and a domestic violence shelter to help female abuse survivors create long-term, liveable wage employment opportunities for women who had been victims of domestic violence. The three organisations began their programme with two basic assumptions (which were integral to their theory of change):
Non-traditional jobs (e.g. electrical, plumbing, carpentry, building management) provide better wages and more opportunities for upward mobility and are more likely to have unions. Therefore, job stability and good wages are more likely if women are trained in these areas.
Women who have been through domestic abuse also need to develop coping skills, workplace behaviours, and have child care available. In addition, they need to be able to manage crises in their lives and such events as court appearances and dealing with the foster care system. If these aspects of their lives are not taken into account, any job training will not likely lead to permanent employment.
The theory of change exercise started by identifying outcomes and assumptions. In this case the long-term outcome was the long-term employment of domestic violence survivors at a liveable wage. To achieve that goal, the programme designers identified three preconditions: survivors attain coping skills, survivors have marketable skills in non-traditional jobs, and survivors know and have appropriate workplace behaviour. The next stage was backward mapping and drawing connections. Once a framework describing the programme was developed indicators for measuring the implementation and effectiveness of the programme were added. The focus then moved to the role of interventions (those things that the programme needed to do to bring about outcomes). Interventions included an outreach campaign, screening, setting up counselling sessions, developing a training curriculum, identifying potential employers and many others. These are represented by the boxed numbers in Figure 3.1. The final stage of the exercise was to write a narrative of the theory of change. The theory is also represented in Figure 3.1.
Figure 3.1 ActKnowledge and Aspen Institute Roundtable on Community Change (2003) Guided Example: Project Superwoman (available at http://www.theoryofchange.org/wp-content/uploads/toco_library/pdf/Superwomen_Example.pdf)
Advantages
There are several advantages to having a TOC in place:
Ownership The TOC provides a unique moment of stakeholder participation where all those who have a stake in the programme can meaningfully contribute to it from the design and conceptualisation stage. Increasing ownership increases commitment and collective synergies and hence the programme’s overall chances of success.
Relevancy Planning with others and with an ear firmly to the ground is hugely helpful in ensuring that the programme meets the needs of its targets in context, and will therefore be relevant.
Focus A TOC planning process begins with a definition of the desired change which means that everything thereof will be defined and decided per reference to this change. The desired change is the focus, and the focus determines the means.
Value for money Programmes that do not think through the elements that a TOC often ‘forces’ a planning process to go through can have less value for money. The lack of focus on the ultimate change may lead to the implementation of multiple activities that in the end are a distraction and do not contribute to generate impact.
Measurement A theory of change can facilitate the measurement and data collection elements of an evaluation (Connell et al. 1998). It helps evaluators ensure that they are measuring the right activities (i.e. the ones that are anticipated to result in change) and that they have developed appropriate research tools. Furthermore, articulating a theory of change at the outset and gaining agreement on it by all stakeholders reduces, but does not eliminate, the problems associated with causal attribution of impact (Connell et al. 1998).
Challenges
We discuss three broad challenges for the use of theories of change: the potential for multiple assumptions about TOC; under-developed TOCs; and the challenge of making sense of causality.
Multiple assumptions
There is not, and neither should there be, anything problematic with different stakeholders bringing different perspectives to bear in the process of developing a theory of change. If anything TOCs are strengthened by this diversity of perceptions that ground projects in its complexity and work with it. Additionally, consensus is not always the reality and power relations permeate all social relations. At some point in this process a choice needs to be made regarding which theory to accept and at this point an agreement needs to be reached.
Instead, the challenge often comes from the different assumptions, and from the difficulty in assessing which of these are critical to the overall success of the initiative. Valters (2014: 10) argues that: ‘Appreciating the difficulties inherent in this task is important, as ignoring them may encourage discussion of arbitrary assumptions or allow people to uncover only those assumptions that they are comfortable defending.’
This poses obvious issues when it comes to evaluating the effectiveness and impact of theory-based projects. Whilst it is possible to validate the assumptions, it may be impossible to confidently say which of the ones that are supported by the evidence would play a greater role in the achievement of the desired change(s).
Under-developed theories of change
Central to the TOC approach, and more so to its ability to be tested and evaluated, are the assumptions. Often these are expected to be substantiated by evidence and in many cases by social analysis. However:
[O]ne potential problem is that Theories of Change can be based on weak and selective evidence bases and build in all kinds of assumptions about the world that are not sufficiently problematised. In this respect they can reinforce and mask the problem they purport to solve, creating a misleading sense of security about the level of critical analysis a programme has been subjected to. (Valters 2014: 4)
This lack of initial analysis will, in turn, affect theory-based evaluations that will be testing assumptions that were not sufficiently problematised and may not even hold as assumptions, let alone as basis for change. In these cases it is right to say that assumptions will be windmills and evaluators Don Quixotes.
Making sense of causality
What makes TOC strong also makes it weak: accounting for and relating complexity will increase the challenge