An Introduction to Evaluation. Chris Fox
as illustrated in Figure 1.1.
However, the underlying assumption in this model is that policymaking is a rational process. The rational model suggests a logical and ordered sequence of policymaking phases that follow a linear process and where evaluation knowledge is apolitical and neutral (Stone et al. 2001). However, many would argue that the reality of policymaking is more complicated. Clearly, policymakers do not just rely on evidence generated from robust evaluations: they bring their own experience, expertise and judgement to the process (Davies 2004). Policymaking also takes place within the context of finite resources and this influences decisions (ibid). Policymakers are also influenced by values (their own, as well as those of politicians and of institutions) along with the habits and traditions of institutions such as Parliament, civil servants and the judiciary (ibid). Outside forces such as lobby groups, pressure groups and consultants are able to influence the policymaking process and that whole process is subject to pragmatics and contingencies such as parliamentary terms and timetables and the capacities of institutions (ibid). Thus many commentators prefer the phrase ‘evidence-informed policy’ (e.g. Treadwell Shine and Bartley 2011).
Figure 1.1 Evaluation through the lifecycle of a policy or programme (source: Public Service Transformation Network 2014, Figure 1.1)
However, some would go further and question the extent to which evidence-based policymaking actually takes place at all. Stone et al. (2001) set out alternative models of policymaking where the role of evidence and of evaluators is either marginal (for example, in a ‘muddling through’ model of policymaking premised on the notions of ‘bounded rationality’ and ‘satisficing’) or only influential at certain stages in the development of a ‘policy paradigm’.
Theory-led evaluation
Donaldson and Lipsey note that:
Reference to theory is widespread in the contemporary evaluation literature, but what is meant by ‘theory’ encompasses a confusing mix of concepts ... A newcomer to evaluation, and even a grizzled veteran, could have a difficult time sorting through the closely related and sometimes interchangeable terms that litter the evaluation landscape ... (Donaldson and Lipsey 2006: 57)
As Donaldson and Lipsey note, the challenge here is partly that terms are used inconsistently and with overlapping meanings, but also partly that the role of theory in evaluation is contentious. They suggest that there are three common types of theory encountered in evaluation:
Evaluation theories are theories of evaluation practice that address questions such as how to understand the nature of what we evaluate, how to assign value to the things we evaluate and their performance, how to construct knowledge, and how to use the knowledge generated by evaluation.
Social science theories attempt to explain the social world, and when they address social phenomena related to social programmes and the social conditions they are intended to improve, those social theories can be very relevant to evaluation.
Theory-based evaluation places emphasis on developing programme theory as an integral component of the evaluation process. It is more modest than social theory and deals with the assumptions that guide the way specific policies, programmes and projects are implemented, and are expected to bring about change.
Evaluation theory
Evaluation theory is largely focused inwards on the evaluation profession and helps evaluators discuss their practice with each other (Donaldson and Lipsey 2006, Shadish 1998). However, evaluation theories can also help evaluators explain various evaluation approaches and practices to those commissioning evaluations, thereby helping find an optimal fit between an evaluation team and the needs and interests of the evaluation commissioner (Donaldson and Lipsey 2006). Alkin and Christie (2004), building on work by Alkin (2004), developed an ‘evaluation theory tree’. They looked at the work of a number of prominent evaluators and placed each of these on one of three branches of a tree designed to indicate the three streams or traditions in evaluation theory. One concentrated on evaluation methods, another on how data were to be judged or valued, and the third on users and use (Mark et al. 2006). The branches represented the evaluator’s relative degree of emphasis (ibid.) on the three issues, and in the graphical representation some evaluators sit close to the junction of two branches while others are located at the far end of a branch. Mark and colleagues suggest the evaluation theory tree is useful in highlighting some of the major conceptual emphases in the field of evaluation. We consider evaluation theory in more detail in Chapter 12.
Social science theory
Donaldson and Lipsey (2006) see several important roles for social science theory in evaluation including informing initial needs assessment and programme design, helping evaluators assess the likelihood that programmes will be able to accomplish certain objectives, and guiding evaluation measurement and design issues. Examples of social science theories given by Donaldson and Lipsey include social cognitive learning theory, the theory of planned behaviour, the theory of health behaviour change, and theories of learned helplessness suggesting that ‘mid-level theories’ are likely to be of most relevance to evaluators.
Theory-based evaluation
Donaldson and Lipsey (2006) suggest that a well-developed programme theory is useful for framing key evaluation questions and designing sensitive and responsive evaluations. This process often involves programme stakeholders in the articulation of programme theory. While most evaluators would agree that a well-developed programme theory is important some groups of evaluators would go further. In recent years both theories of change and realistic evaluation have developed rapidly and they possess some similarities: both see limitations in methods-driven approaches to evaluation design, see theory development as critical to the evaluation process, and emphasise the importance of programme context in understanding how programmes lead to changes in outcomes (Blamey and Mackenzie 2007). And yet they are not entirely interchangeable approaches, use theory in slightly different ways, and arguably are differently suited to evaluating projects, programmes and policies (ibid.). In Chapter 3 we look in more detail at programme theory and in particular at a popular evaluation tool – ‘theories of change’.
Trends in evaluation
Mark et al. (2006), in their brief commentary on the history of evaluation, argue that there is not a single history of evaluation but instead multiple histories of evaluation depending upon one’s discipline and domain of work. Below, we look at trends in evaluation through a number of different lenses.
Evaluation ‘booms’
Evaluation has a relatively short history when compared to that of the wider social sciences. Some histories of evaluation, particularly with a US focus, point to the evaluations of social programmes in sectors such as education and public health prior to the First World War as the start of the modern history of evaluation, characterised by the application of social research methods to programme evaluation (see, for instance, Rossi et al. 2004). By the 1930s, social scientists were using rigorous methods to assess a range of social programmes, with the Second World War providing a strong boost as evaluators worked with the US Army to develop procedures for monitoring soldier morale and evaluating personnel policies and propaganda techniques (ibid.).
The first ‘boom’ in evaluation
Donaldson and Lipsey (2006) note that the first major boom in evaluation seemed to occur in the United States in the late 1960s and 70s under the Kennedy and Johnson administrations, when large social programmes funded by Federal government led to the development of experimental and quasi-experimental