An Introduction to Evaluation. Chris Fox
influential in this movement. Many policymakers and academics from across the social policy spectrum and on both sides of the Atlantic have been impressed with the Cochrane Collaboration (www.cochrane.org) and its impact on health care, and have discussed the possibility of implementing similar models of evidence-based practice in sectors as diverse as criminal justice, education and development. The more recently established Campbell Collaboration (www.campbellcollaboration.org) is perhaps the most systematic in attempting to replicate this approach across social policy.
Chapter Summary
Evaluation can be distinguished from monitoring, performance management, audit and accreditation by its use of social research methods.
Evaluation can be distinguished from other types of research by its emphasis on judgements of value. Research is undertaken to resolve a problem while evaluation is undertaken to establish value. Evaluation is thus inherently political, raising questions about whether it is necessarily subjective, or can claim objectivity.
A formative evaluation studies the development of a programme.
A summative evaluation studies the effectiveness of a programme.
A process or implementation evaluation studies the process of delivering a programme.
An outcome or impact evaluation studies the outcome of a programme.
An economic evaluation studies the worth of a programme.
Theory-based evaluations start with a programme theory that informs the evaluation design and subsequent evaluation activity.
Evaluation has a relatively short history when compared to the wider social sciences.
One way to understand the historical development of evaluation is to set it alongside the development of government. Evaluation had its first ‘boom’ in the post-war period as governments expanded their health and social responsibilities. Over recent decades trends in evaluation have reflected neo-liberal and managerialist trends in government.
Some would argue that the most recent ‘wave’ of evaluation is inextricably linked to the move towards ‘evidence-based policy’, and is characterised by the resurgence of a ‘scientific’ approach to evaluation.
Further Reading
The introductory chapter to The SAGE Handbook of Evaluation by Mark and colleagues discusses definitions of evaluation and provides a short overview of the history of evaluation:
Mark M, Greene J and Shaw I (2006) ‘The evaluation of policies, programs, and practices’, in Shaw I, Greene J and Mark M (eds) The SAGE Handbook of Evaluation. London: Sage.
Similar ground is covered by Rossi and colleagues in the first chapter of their book:
Rossi P, Lipsey M and Freeman H (2004) Evaluation: A Systematic Approach. London: Sage.
While the focus of the article is on dissemination, Vedung also provides a concise view of how evaluation has developed as a discipline and some of the big ideas that have shaped it:
Vedung E (2010) ‘Four waves of evaluation diffusion’, Evaluation 16(3): 263—77.
1. Arguably, this distinction in products is diminishing as researchers and research funders place more emphasis on research achieving ‘impact’ and research knowledge being ‘mobilised’ more effectively.
2 The Ethics of Evaluation
Introduction 24
Ethics in evaluation and other forms of social research distinguished 24
Ethical guidelines 25
The guiding principles 26
Limitations of ethical guidelines 31
The challenges that evaluators must overcome to ensure ethical evaluations 33
Introduction
‘Ethics is about how we behave or should behave as individuals and as part of the society in which we live in interaction with others’ (Simons 2006). Evaluation is a relatively new discipline and it is only quite recently that ethical considerations and the structures and processes to address them have become prominent in the sector.
In the early pages of their chapter on the ethics of social research, Hesse-Biber and Leavy state that ‘[T]he history of the development of the field of ethics in research, unfortunately, has largely been built on egregious and disastrous breaches of humane ethical values’ (2010: 60). Their example is a public health study of untreated cases of latent syphilis in human subjects in the context of which, despite the existing scientific knowledge at the start of the study and of the subsequent scientific advances, participating patients were observed on the basis of an outdated therapy.
Particularly after the Second World War, as biomedical research became increasingly regulated from an ethical point of view, ethical issues in social and to an extent psychological research remained largely ignored. When the ethics in social and psychological research began being addressed more systematically, the onus was on the treatment of the research ‘subjects’ – research ethics and researchers’ ethical behaviour were determined by how the ‘subjects’ of the research would be treated.
Ethical considerations in evaluations came even later and a few evaluation societies have led on this. For example, the American Evaluation Association’s Guiding Principles for Evaluators were adopted by the Association in 1994 (Simons 2006).
In this chapter, we begin by distinguishing ethics in evaluation from ethical considerations in other forms of social research. We then take a brief look at some of the ethical guidelines for evaluation that have been published and the role of ethics committees in evaluation. The main part of this chapter considers the ethics inherent to the design and implementation of evaluations as well as the evaluation practices that in some key sectors may pose ethical risks. We set out guiding principles that relate to participants’ and stakeholders’ rights and should govern evaluators’ inherent ethics and ethical behaviour. We also discuss the limitations to ethical guidelines.
We next look at challenges to ensuring ethical evaluations, considering two types of challenge: those relating to the implementation of certain methods and those relating to exogenous factors.
Ethics in evaluation and other forms of social research distinguished
There are some key differences between ethics in evaluation and ethics in social research more generally. For Simons (2006), these stem in part from the characteristics of the evaluation field that distinguish it from other types of social research, including the public nature of evaluation in providing an evidence base for informed judgement and public dialogue, the centrality of judgement in the evaluative process, the inherently political nature of evaluation, and the need for evaluation to be independent. Also important however is the context within which much evaluation takes place, particularly the ‘intensification of accountability and managerialism in institutions, an increase in governance and quality assurance programmes, and a preoccupation with audit, monitoring, regulation and control’ (Simons 2006: 244). Simon suggests that a greater emphasis on ethics in evaluation may, in part, be a response to this external pressure for regulation.
One notable difference lies in the stated need for the evaluator to behave ethically (or of ‘being principled’) independent of the evaluation’s characteristics, focus, target audience, and so forth. In research more broadly researchers need to ensure that certain ethical guidelines are followed to create an ethical environment for the ‘subjects’ of the research