An Introduction to Evaluation. Chris Fox
skill sets from project planning, team building and stakeholder management to the design of interdisciplinary and multi-method research instruments. It is this diversity and the applied nature of evaluation research which make it an exciting and stimulating task.
Good evaluators are not just expert social scientists, they are also expert negotiators and project managers. They have a deep knowledge of social science method and theory, but are also practical and politically astute.
Our approach to evaluation
A practical discipline
Methodological rigour, theory development and a coherent philosophy are all important in evaluation, but above all it remains a practical discipline. Academic debate about the merits of different evaluation paradigms, types of data collection or approaches to analysis must be tempered by the needs of different stakeholders, including funders, policymakers and service users. Unlike some other types of social science research, evaluators almost always work within practical constraints such as fixed budgets and deadlines.
Evidence-based (or informed) policy and practice
One of the distinguishing features of evaluation is that it is undertaken to improve programme effectiveness and/or inform decisions about future programmes (Patton 1997). However, this raises lots of challenges and questions for evaluation, including the competing needs and interests of various stakeholders and the extent to which evidence can or should influence policy and/or practice. Effective planning, good project management and a proper strategy for knowledge mobilisation are just as important as a well-designed evaluation framework and the appropriate use of research methods.
Multi-sector, multi-disciplinary
Evaluation has an established or growing profile across a wide range of sectors including education, welfare, health and social care, criminal justice and international development. Different approaches to evaluation are prominent in different sectors. In writing this book we have tried to draw on examples from a range of sectors in order to illustrate the range and diversity of evaluation theory and practice and encourage cross-pollination of ideas.
Complexity
Possibly the biggest challenge facing evaluators today is how to respond to complexity. We think of complexity in two senses.
Firstly, across a number of sectors there is increasing recognition that service users often have multiple and ‘complex’ needs and that services often have to be organised that involve collaboration between various organisations. Here we are recognising that people and programmes are complicated. So, for example, a housing need may be linked to a substance misuse need that is, in turn, linked to a mental health need. Alternatively, an individual child’s educational needs may be interlinked with a broader set of health and social care needs within their family and those needs may be intergenerational. These needs require multiple delivery partners working across different sectors and service delivery often takes place within a fast-moving policy environment where change is the norm.
Secondly, we also think of complexity in a sociological sense. This goes beyond the idea that programmes are complicated and embraces the concept of complexity (Glouberman and Zimmerman 2002). Complexity includes the principle of non-linearity (small changes in inputs may, under some conditions but not others, produce large changes in outcome); the contribution of local ‘adaptiveness’ and feedback loops; the phenomenon of emergence; the importance of path dependence; and the role of human agency (Marchal et al. 2013).
In both of these senses, complexity poses challenges for evaluators seeking to explain how interventions work, what impacts they have, and whether they make economic sense.
What does this book cover?
This book includes:
practical steps for designing and conducting an evaluation and ensuring that the evaluation is ethical
different approaches to evaluation, including theories of change, process evaluation, impact evaluation (outcome evaluation) and economic evaluation
ways of reviewing evaluation evidence including systematic reviews
how to get evaluation knowledge into policy and practice
important philosophical questions about evaluation and its role in society
The book is divided into several sections.
Part I: Getting Started
This section introduces evaluation, providing a brief overview of the field and highlighting some key issues to address before staring an evaluation.
Chapter 1 introduces the field of evaluation and provides an overview of different types of evaluation and the historical development of the discipline.
Chapter 2 considers the ethics of evaluation, both the design of ethical evaluations and ethical evaluation practice.
Part II: Undertaking an Evaluation
This section covers common components of an evaluation. Not every evaluation will use all these components but most evaluations will use some of them.
Chapter 3 introduces Theories of Change, an important tool for evaluators and programme developers and an approach that can be integrated into many evaluations.
Chapter 4 discusses process (implementation) evaluation.
Chapter 5 discusses impact (outcome) evaluation.
Chapter 6 discusses economic evaluation.
Chapter 7 introduces a range of quantitative and qualitative methods for data collection and analysis commonly used in evaluation. Mixed methods (dual strategies) are also considered.
Part III: The Practice of Evaluation
This section provides practical guidance on the practice of evaluation.
Chapter 8 gives practical guidance and advice on planning an evaluation, with a particular focus on refining evaluation questions and engaging with different stakeholder groups.
Chapter 9 provides practical guidance and advice on conducting an evaluation, with a particular emphasis on the importance of good project management.
Part IV: Using Evaluation Findings
Over recent years, increasing emphasis has been placed on the use of evaluation findings.
Chapter 10 describes systematic reviews, which are used increasingly to synthesise evaluation findings for policymakers and practitioners.
Chapter 11 gives practice guidance on disseminating evaluation results and also discusses how to include evaluation evidence in policy and practice (knowledge mobilisation).
Part V: Evaluation Paradigms