An Introduction to Evaluation. Chris Fox
the emphasis on causal attribution has been downplayed over the years, and particularly in the performance evaluation of public sector initiatives in the context of a mainstreamed managing for results approach, where it is accepted that ‘the manager might be only one of several players trying, with the resources and authorities available, to influence the achievement of the intended outcomes’ (Mayne 2001: 2).
Gradually, therefore, the emphasis has been shifted to contribution or the attempt to demonstrate plausible association as opposed to bulletproof, beyond reasonable doubt attribution. This distinction between attribution and contribution is essential:
In much of the literature, attribution is used to both identify with finding the cause of an effect and with estimating quantitatively how much of the effect is due to the intervention. The term contribution is used here in the following way: in light of the multiple factors influencing a result, has the intervention made a noticeable contribution to an observed result and in what way? (Mayne 2012: 273)
The acceptance of plausible association or contribution suits theory-based evaluations (Mayne 2012) and to a large extent helps deal with the challenge of complexity and multiple contributing factors.
Theory-based evaluations
According to Connell and Kubisch (1998), a TOC should be the following:
Plausible There must be available evidence that sustains the assumptions, and hence will support the change potential of the activities to be implemented.
Doable The necessary resources – from financial to intuitional – must be in place to ensure that the TOC informed initiative can be operationalised.
Testable It must be specific and complete enough for the evaluator to assess progress and evaluate the contribution to change.
The emphasis of the TOC is on the social change that one wants to enable. As an approach the TOC’s aim is to arrive at a measurable description of this change, and this is the link between TOC and evaluation. Particularly in interventions that in their planning used the TOC approach, evaluations can be utilised to assess the:
validity of the assumptions in context
pertinence of the pathways of change
effectiveness of the strategies
achievement and impact of the change
Theory-based evaluations are either carried out on programmes that have theories of change in place, or begin by trying to reconstruct the theory of change of a programme in order to increase the evaluation’s focus. In both cases, evaluations test the theory behind the intervention to better understand what worked, how this came about, and whether the change(s) had been achieved. Commonly, these set out to undertake the ‘analysis and valuation of the contribution of intervention strategies to resolving or controlling social problems’ (van der Knaap 2004: 17).
There has been a lot of theorising around theory-based evaluations. The plethora of definitions on offer can be such that some authors opt to come up with their own (and yet another) definition. Stame (2004) makes a useful distinction between theory-driven, theory-based and realistic evaluations:
Theory-driven evaluations apply to so-called ‘black box’ programmes (Chen and Rossi 1989) or programmes without theory. In these cases, evaluations should provide these programmes with a sound social theory to underpin its original ambitions and goals. The effort is therefore to understand the programmes’ inner-logics, and less emphasis is put on understanding the wider social context and its influence.
Theory-based evaluations place a greater emphasis on context than is the case with theory-driven evaluations. Evaluators should therefore work under the assumption that programmes’ theories of change are also evolving with the context and are influenced by it (Weiss 1987) and context ‘means the social, political and economic setting in which the programme takes place, all of which can influence how the causal chain plays out’ (White 2009: 11). Context is also highly political and politicised. Hence, ‘[G]ood programme theory is not theory that is unaffected by politics, but a good theory of how politics works in practice’ (Stame 2004: 61). Additionally, theory-based evaluations must take into account that TOCs ‘have two components: “implementation theory”, which forecasts in a descriptive way the steps to be taken in the implementation of the programme; and “programmatic theory”, based on the mechanisms that make things happen’, and therefore the aim of theory-based evaluations should be to ‘make these mechanisms clear, break down a programme in its subsequent mechanisms, and use data of different kinds to test them’ (ibid.: 62).
Realistic evaluations try to relate the context, mechanisms (of implementation) and outcomes (Pawson and Tilley 1997). Context is particularly important and ‘[R]ealist approaches assume that nothing works everywhere or for everyone, and that context really does make a difference to programme outcomes’ (Westhorp 2014: 4). Change (i.e. outcomes) is generated by people in their process of interacting with the intervention in context and being exposed to it. Understanding how people chose and use the mechanisms of implementation available to them in context to generate change is one of the central objectives of realistic evaluations. Hence programmes are much more fluid and evolving and it is ‘the interaction between what the programme provides and the reasoning of its intended target population that causes the outcomes’ (ibid.: 5). Accordingly, within a realistic evaluation approach, ‘the evaluator needs to identify what resources, opportunities or constraints were in fact provided, and to whom; and what “reasoning” was prompted in response, generating what changes in behaviour, which in turn generate what outcomes’ (ibid.: 5). Ultimately, ‘policy-makers and practitioners need to understand how and why programmes work and don’t work in different contexts, so that they are better equipped to make decisions about which programmes or policies to use and how to adapt them to local contexts’ (ibid.: 4).
From the above, it is possible to identify what is common between the three different evaluation approaches:
The programme theory: in all three cases the evaluation either starts with the programme theory, or in the absence of one begins by reconstructing this retrospectively.
Context is important, albeit to different degrees in the various approaches, particularly because of the way context influences the programme, its stakeholders, and hence its theory.
Feedback from programme stakeholders, and procuring this feedback, are essential for understanding and reconstructing the theory underlying the programme. Indeed, ‘despite the fact that rationality is central to the notion of theory in public policy, contextuality and subjective judgements are never dismissed as mere methodological flaws: rather they form the framework in which policy theories can be useful, discussed and valued in the first place’ (van der Knaap 2004: 17).
Evaluation focuses on the interface between the conceptual (the casual or change model, the assumptions) and the empirical (testing the programme’s contribution to the change or changes).
The theory determines the method, hence theory-based evaluations are not method-prescriptive. Essentially, ‘[N]o method is seen as the ‘gold standard’. Theories should be made explicit, and the evaluation steps should be built around them: by elaborating on assumptions; revealing causal chains; and engaging all concerned parties in this exercise’ (Stame 2004: 60).
Chapter Summary
The TOC was first developed as an evaluation tool to use in complex social programmes and it evolved into a useful planning and evaluation approach.
Planning-wise, the TOC consists of stating the desired (long-term) change based on a number of assumptions that hypothesise, project or calculate how change can be enabled.
As an approach the TOC’s aim is to arrive at a measurable description of the expected change, and this is the link between TOC and evaluation.
Theory-based