An Introduction to Evaluation. Chris Fox
paths are summarised in logical frames (logframes), a supportive tool for the approach.
Originally adopted by the US Agency for International Development in the mid-1960s for widespread project management, the logical framework approach reflected a greater push to professionalise the aid sector by embedding its programmes in a management logic. The approach was quickly adopted by other agencies.
Especially in the development sector, at some point very few in the donor community were willing to fund projects that did not incorporate this approach. For donors, the logical framework approach was essential for any evaluation of the funded interventions given that it laid out the logic that the evaluations were due to test:
The logical framework is not itself an evaluation device; rather, it sets the stage for the evaluation. Evaluation consists of determining and validating whether or not the project outputs were produced, whether these outputs in fact achieved the project purpose, and finally whether this achievement made a significant contribution, as planned, to the higher goal. By focusing on the causative linkages between inputs, outputs, purpose, and goal, evaluation avoids extraneous and irrelevant questions and looks for possible improvements. (USAID 1974: 14–15)
Although some would argue that in its beginnings the logical framework approach was developed as a sort of theory of change (e.g. Vogel 2012), there are fundamental differences between the two that are rooted in the type of thinking that supports each one.
In the theory of change approach, the beginning is the end – the change or changes we want to see. From there we build backwards, critically assessing the context and its influence over the different strategies we may use and the people we want to target. Indeed, TOC ‘turns conventional planning on its head because it pushes groups to first work out their goals or desired impact and work backwards on outcome pathways rather than engage in conventional forward oriented “so-that” reasoning’ (Taplin et al. 2013: 3). This interaction between context, people, strategies, resources, and so forth effectively accounts for uncertainty and deals with complexity.
The logical framework approach is narrower in that the focus on the external context is considerably less than the focus on the programme’s goals and objectives. Almost part of the logical framework approach’s DNA is this post-war management optimism that context is ‘controllable’ (Chambers 1997; Hersoug 1996) and external influences can be managed just like any other project-related variable. Yet, the reality is different and in the ‘real’ world, where projects take place:
A single centre of authority rarely exists and – in practice – clearly agreed to objectives are rare to find. When pursuing these objectives in situations in which there are multiple and diverse stakeholders or changing or uncertain context conditions, the LFA tends to over-specify objectives and to over-emphasize control as opposed to flexibility. (Hummelbrunner 2010: 3)
More specifically, as Gasper put it, the logical framework approach:
attempts something extraordinary: management of public, indeed international, programmes by detailed agreed objectives. It derives from work in engineering, military and private business contexts, whereas in many public, developmental and inter-organizational contexts we lack strong knowledge of causal links and a simple authority setup or its substitute, a shared vision. (Gasper 2000: 3)
In sum, the logical framework approach reflects a style of ‘management by objectives’ that makes assumptions about actors having and using all relevant information in a stable environment that is seldom realistic (Hummelbrunner 2010). This, of course, poses considerable challenges to use of the logical framework both as a management tool and as a basis for programme evaluation. One difficulty occurs where an evaluation framework is based on a logframe prepared during early project planning. FASID (2010) notes that by the end of the project the logframe may no longer embody the project, hence posing challenges for an ex-post evaluation based on it.
The TOC on the other hand works with and accounts for context and the potential of its non-predictable nature. The weight of activities for instance is downplayed; these might change to adapt to the context or may be changed by stakeholders in their interaction with context. A project is not what one does but what one wants to achieve. In a changing context, what one does, how one does it and when may also have to be altered in order for the desired change to be achieved.
Additionally, the TOC reflects a constructivist stand by accepting that to understand a context one needs to understand stakeholders’ contextual perceptions. Stakeholders will interact with the context based on their perceptions of it; the project will be part of the context and therefore of stakeholders’ perceptions as well. This also ensures that the TOC accounts for different cultural realities and interpretations. The logical framework approach does have a more positivist aspect in that the context is often expected to be evidenced and expertly validated.
Particularly in the international development and aid sector, the TOC has gained ground. This is mostly because the TOC ‘represents an increased desire for organisations to be able to explore and represent change in a way that reflects a complex and systemic understanding of development’ (Stein and Valters 2012: 3). This ability to think through complexity and translate it into a project is what makes the TOC approach attractive to other sectors as well.
Yet this is mostly reduced to the dominant discourse. In practice, most programmes are solely defined following a logical framework approach or simply use a logframe (Bakewell and Garbutt 2005). This seems to be the easiest option and more so when donors are also not demanding anything over and beyond logframes. It can indeed be argued that in many cases and in many sectors:
implementing organisations are rarely incentivised to present the complexity of their interventions to donors, since admitting the unpredictability of change through an intervention doesn’t inspire confidence that the programme will get ‘results’ – and in turn get them funding – however realistic and honest that assessment is. Donors and implementers alike have to negotiate the political context in which they work, which may also constrain the honesty of their analysis of their own and others’ policy and practice. This is the context in which the ‘Theory of Change’ approach has rapidly emerged as a new management tool in the aid industry. (Valters 2014: 1)
How to develop a TOC
As mentioned above, the TOC of a programme can either be developed before the start of the programme (prospective) or it can be ‘reconstructed’ by the evaluator (retrospective). The different situations dictate the way in which the TOC will be applied and used.
Prospectively: TOC as an approach to planning
When the TOC is used as approach to planning, participation is key. It all starts with getting the main stakeholders together to discuss and agree on the issue (problem) and ambition (change). As Taplin et al. put it:
An important first step in the process is identifying a workable long-term goal and outcomes. The long-term outcome should be something the initiative can realistically achieve and that everyone involved understands. (Taplin et al. 2013: 3)
Table 3.1 illustrates the necessary steps.
Discussions do need to lead to a comfortable agreement given that in organisational contexts in the absence of a minimum common ground programmes may fail. In these cases, gathering a minimum of agreement implies recognising the differences and agreeing that these are not fundamental so as to prevent collective energy and synergy around the programme.
Table 3.1
Retrospectively: TOC as part of ex post evaluation
Reconstructing a programme’s TOC does involve a lot of work. The evaluator may start with programme documentation such as funding bids, project plans or steering group minutes. Often the evaluator needs to conduct a series of structured and semi-structured interviews with key informants and stakeholders to piece