An Introduction to Evaluation. Chris Fox
are considered, planned for and reviewed by an appropriate ethics committee or similar.
Chapter Summary
Ethics is about how we behave or should behave as individuals and as part of the society in which we live in interaction with others.
There are many professional societies, associations and committees which publish guidelines on ethical evaluation and scrutinise the ethical dimensions of evaluations.
Commonly accepted principles relate to participants’ and stakeholders’ rights: informed consent; voluntary participation; do no harm; and confidentiality and anonymity.
Evaluators’ ethical behaviour should be governed by concerns for professional integrity, openness and respect for diversity, and cultural competency.
Criticisms of ethical guidelines include the accusation that commonly accepted guidelines for ethical practice are ‘Western-centric’ and designed primarily to protect the interests of the evaluation profession.
Evaluators need to overcome the challenges posed by particular methodological approaches – for instance the use of randomisation or participatory action research –and factors outside the evaluator’s control such as power relations and intercultural communication.
Further Reading
Further reading will depend on the country and sector within which the reader is located. Many countries have an evaluation society or professional body that publishes evaluation guidelines which readers may wish to familiarise themselves with. Some sectors have specific evaluation guidelines and processes that an evaluator should abide by.
Stake and Mabry (1998) are a good resource on ethics as applied to programme evaluations, with a focus on the behaviour of evaluators as opposed to looking at ethics solely as a series of tick-the-box guidelines:
Stake R and Mabry L (1998) ‘Ethics in program evaluation’, Scandinavian Journal of Social Welfare 7(2): 99—109.
Chapter 9 of Robson (2011) provides a good introduction to ethics in applied research:
Robson C (2011) Real World Research (3rd edn). Chichester: Wiley.
3 Theories of Change
Introduction 42
What is a theory of change? 42
How to develop a TOC 49
Advantages 53
Challenges 53
Theory-based evaluations 55
Introduction
How does change happen, therein lies the question – one that led to the development of the theory of change (TOC) approach. Developed in the 1990s at the Aspen Institute Roundtable on Community Change, the TOC was first articulated as a tool to evaluate complex programmes. At the heart of this development was the recognition that planning is important for a programme evaluation and the TOC emerged as a tool to attempt to fill this void. Subsequently the TOC has been used by programme developers as a programme planning tool.
A recognition that programmes are based on theory inspired the TOC approach. TOC however represents one step further in that it understands change is not always logical and emerging from neatly organised casual chain(s) of events.
Nowadays, one can say that the TOC is an approach to programme evaluation and programme planning. It has become common practice to start an evaluation with an elaboration of the TOC. For evaluators this has several advantages. Firstly, it helps evaluators ensure that their subsequent evaluation design ‘fits’ the programme they are evaluating. Secondly, it can improve the quality of communication with evaluation stakeholders. Thirdly, the TOC can be helpful when discussing the aims and objectives of the evaluation with funders and making sure that their expectations are realistic. Those who are developing programmes will often use the TOC approach to ensure that their programme is logically consistent.
The TOC approach normally starts by stating the desired change based on a number of assumptions that hypothesise, project or calculate how that change can be enabled. The assumptions therefore determine the strategies (activities) that can be used to achieve the change(s). The focus of theory-based evaluations is mostly on the evaluation of the assumptions. Assessing whether or not these are/were correct will lead to a greater understand of (a) whether change happened and (b) how.
This chapter will begin with the origins of the TOC, its main characteristics, what differentiates the theory of change approach from the logical framework approach, and how best to develop a TOC. After going over the advantages and challenges of using this approach, the chapter ends with a section on theory-based evaluations – what they are, the different types, and a description of these.
What is a theory of change?
Origins and definition of Theory of Change
The theory of change was fully articulated in the 1990s at the Aspen Institute Roundtable on Community Change. This articulation came out of a discussion regarding the evaluation of complex programmes, namely those on community initiatives aiming for social and political changes. Briefly, these initiatives:
indicate a commitment to change at many levels, including individual, family, institutional, and community-wide, through processes that involve collaboration and coordination within the community and between the community and the broader society. (Kubisch et al. 1998: 2)
Evaluating Complex Community Initiatives (CCIs) was found to be challenging (Kubisch et al. 1998) due to the following:
Horizontal complexity The interlinkages and interconnections between different variables and factors pose questions regarding what to measure. These initiatives may act on several fronts, trying to tackle diverse issues at the same time, e.g. improving housing, employment skills, and relationships within the family. Clearly discerning what has worked and what has not is a challenge.
Vertical complexity In CCIs interventions aim at different levels – community, institutional, family, and individual. Again, and because of the interlinkages, it becomes very difficult to know at what level change is having a ripple effect.
Contextual issues Macro-level and structural issues are particularly challenging, e.g. economic environment and ethnicity. In these cases, it may be almost impossible to act on the barriers to further improvements and change.
Flexible and evolving intervention By design, CCIs are meant to be malleable and adaptable to community dynamics and its organic development. This offers its own challenges when it comes to monitoring these initiatives and assessing its impact.
A broad range of outcomes These are are mostly at different levels. The horizontal and vertical complexity of these initiatives leads to the formulation of different goals for different areas on intervention that may sit at different levels. The definition, understanding, and hence operationalisation and measurement of these goals are challenging.
Absence of a comparison community or control group In most cases, in its inception there are no provisions to ensure that a comparison community or control group is followed in order to robustly assess the impact of the initiative and its contribution beyond reasonable doubt.
Weiss (2000) hypothesises that a key reason why CCIs and other complex programmes are difficult to evaluate is that the theories of change that underpin them are poorly articulated. The theory of change is essential nevertheless in that it relates the programme theory and the implementation theory. It posits questions that trigger a reasoning which starts with the change, the assumptions behind what makes the change happen