An Introduction to Evaluation. Chris Fox
that involve the use of control groups, and in the case of experiments a random allocation of evaluation subjects between an intervention and a control group. They are designs that draw on the tradition of laboratory experiments in disciplines such as medicine, psychology and agriculture. We discuss these evaluation designs in detail in Chapter 5. This first, post-war, boom period was also characterised by developments in social research methods, including sample surveys and advanced statistical procedures (Rossi et al. 2004). An evaluation community was emerging by the 1970s, and the first journal in evaluation, Evaluation Review, was launched in 1976 by Sage Publications (ibid.).
The second ‘boom’ in evaluation
Building on their previous work, Donaldson and Lipsey (2006) describe a ‘second boom in evaluation’ which happened during the last years of the twentieth century and the early years of the twenty first. This is more global and includes a wider range of governmental and non-governmental organisations commissioning evaluations and a growing number of evaluation professionals and professional associations. Evaluation as a practice has become globalised principally due to the link between effectiveness, transparency and accountability. Another characteristic of this second boom is the development of new theories of evaluation, new evaluation methods and new evaluation tools designed to address a broader and more diverse range of evaluation practice challenges (Donaldson and Lipsey 2006). This also represents a shift with evaluation, no longer shaped mainly by the interests of evaluators and consumers of evaluation exercising significant influence (Rossi et al. 2004). For Rossi and colleagues the incorporation of the consumer perspective has moved evaluation beyond academic social science:
Evaluation has now become a political and managerial activity that makes significant input into the complex mosaic from which emerge policy decisions and resources for starting, enlarging, changing or sustaining programs to better the human condition. (Rossi et al. 2004: 10)
An implication of this move is that evaluation must be seen as an integral part of the social policy and public administration.
Real time evaluation
Real time evaluation (RTE) as a concept and practice took greater hold in the humanitarian sector. This was driven by a need to produce rapid assessments of interventions in emergency situations, where time is of the essence and failures are not opportunities for learning. Rather, programme failures in emergency contexts may lead to dramatic consequences. The focus is mostly on process. Additionally:
Unlike the majority of final ex-post evaluations, the process and products of an RTE are integrated within the programme cycle. Interaction with programme staff and managers during the course of implementation means that discussion, which may or may not be reflected in a final document, can help to bring about changes in the programme, rather than just reflecting on its quality after the event. (Herson and Mitchell 2005: 43)
More recently, Oxfam began trialling real time evaluations in its interventions in fragile and conflict-affected states and in crisis situations, i.e. in rapidly changing contexts that were unstructured and unpredictable, much as in emergency situations.
RTE embodies the rapid metamorphosis that evaluations go through notably in certain sectors like the humanitarian and international development ones. The latter are increasingly under pressure to demonstrate effectiveness and impact due to either the squeeze in countries’ Official Development Assistance (ODA) or change in the prioritisation of countries’ ODA allocations. However demonstrating effectiveness and impact in complex, rapidly changing contexts, where the availability of and access to data can be a challenge, has led to a proliferation of evaluation approaches that do not strictly conform to textbook instructions.
Evaluation, social policy and public administration
Several commentators have charted the relationship between the evaluation sector and changing fashions in government.
The growth of government
Rossi et al. (2004) note that the expansion of evaluation accompanied the (US) expansion of government, particularly at the federal level. They note that in the 1930s the Great Depression in the US saw rapid growth in ‘human services’ provided by government, and during that same period social scientists started to investigate the political, organisational and administrative decision making that took place in government. As government became increasingly complex and technical the importance of evaluation started to be acknowledged by politicians and administrators (ibid.). As described above, evaluation continued to expand rapidly during the Kennedy and Johnson administrations in the 1960s in response to large-scale social programmes such as the War on Poverty and the Great Society (ibid.). This was growth primarily in summative, goals-based evaluation – what Vedung (2010) describes as the ‘science-driven wave’:
From its inception in the late 1950s and consolidation in the mid-1960s, evaluation has been embedded into one of the great narratives of our time: that the world can be made more humane if capitalism and the market economy can be reined in by appropriate doses of central policy planning and public intervention at a comprehensive level ... In public-sector thinking, this was hailed as a victory of a kind of rationality. Public policy should be made more scientific and sensible. (Vedung 2010: 265)
Neo-liberalism and new public management
However, in the post-war period the dominance of the scientific approach to evaluation most clearly expressed in experimental evaluation designs was challenged on a number of fronts. In some branches of the social sciences the positivist paradigm that the scientific approach rests upon was questioned (we discuss this in more detail in Chapter 12) and throughout the 1970s and 80s evaluation theory saw the increasing influence of more pluralistic and naturalistic approaches to evaluation ( Lincoln and Guba 1986). In the US, the Reagan administration of 1980 signalled the start of attempts to curtail domestic federal expenditure (Rossi et al. 2004), and in the UK Thatcher’s government of 1979 introduced a new approach to public sector accountability that since has become known as ‘the new public management’ or ‘managerialism’ (Palfrey et al. 2012), a trend that continued through successive governments including the Blair administration of 1997 (6 and Peck 2004), the Brown administration and the Conservative government under the stewardship of David Cameron. This new emphasis on accountability and ‘value for money’, or what Vedung refers to as the ‘neo-liberal wave’, has, if anything, strengthened the position of evaluation, although at the same time it has changed it:
In the neo-liberal wave, it is regarded as imperative that the fundamental principal in a representative democracy, the demos, has a right to know how her agents spend her money. This results in an increased emphasis on the accountability of agents in terms of resource use, by checking for economy, effectiveness and cost efficiency. Evaluation has thus been strengthened and, above all, taken on new forms. Evaluation has become a permanent feature of results-based management and of outsourcing. Evaluation has taken on new expressions in the form of accountability assessments, performance measurement and consumer satisfaction appraisal. Quality assurance and benchmarking are also recommended. (Vedung 2010: 273)
Evidence-based policy and practice
However, it is too simplistic to characterise the development of the evaluation sector as starting with goal-based models, which are then superceded by actor models and economic models. In this regard, Vedung’s (2010) analogy of different waves of evaluation depositing sediments which form present-day evaluation activities is useful. For Vedung the most recent ‘wave’ of evaluation started in the mid-1990s and is the ‘evidence-based wave’. In this wave, a hierarchy of evaluation designs in which experiments are the ‘gold standard’ has been advocated across a number of different sectors and championed by governments of differing political persuasions on both sides of the Atlantic. There is no doubt that the rise of Evidence-Based Medicine, in which systematic