An Introduction to Evaluation. Chris Fox
in theory all of this seems very straightforward, i.e. that any conscious and well-informed individual is able to opt in or out of actively participating in the evaluations, in practice challenges to the concept of voluntary participation may arise.
Evaluations pose issues of power relations that are of a different nature from those in social research, and more specifically programme evaluations. It is not simply about the power relations between the evaluator and participants but the complex power relations between participants and those who have commissioned the evaluations.
In some circumstances, individuals may find it difficult to opt out when they are being told by their managers that they have to participate. Evaluators are unable to control this and are largely unable to influence the interactional dynamics underlining organisational hierarchies and complexity. They must however be highly aware of it and ensure that they ask participants directly whether they want to participant or not as opposed to asking those who manage them.
Indeed, in the overwhelming majority of cases, evaluators do need to invest more effort in ensuring that participants: (a) are fully aware of and understand the main purpose, objectives, and other relevant details concerning the evaluations; (b) understand what are the advantages and disadvantages associated with their decision to participate or opt out; (c) are given the opportunity to negotiate their participation; and (d) are able to opt out at a later point if they change their mind.
A particular challenge is posed by the use of incentives or payments. Sometimes a payment or similar incentive (for example a voucher) is deemed necessary or appropriate to compensate participants for giving up their time or to demonstrate appreciation of the participants’ input. Careful consideration must be given to the influence of the incentive on response rates and responses, with the level of incentive set so as to minimise the risk of introducing a bias to the research.
Do no harm
Generally, this means that participants, contributors and evaluation stakeholders more broadly will incur no harm if and when they decide to participate. This principle needs to be mainstreamed throughout and influence the overall evaluation rationale and process – from the selection of the methods to the actual implementation of these. In short, the evaluator must ensure that the evaluation by its design and implementation follows the principle of ‘do no harm’.
It is necessary to ensure that participants are not put through any personal duress and that in no circumstance are they to have their safety compromised. Personal duress can emerge from situations where participants are asked personal questions that put them in an uncomfortable position (e.g. health evaluations, evaluations that require interviewing rape victims or the victims of violent conflicts).
The safety of participants can be easily compromised, particularly in cases where evaluators do not think more broadly about these issues. In some contexts the very fact that participants are seen talking to the evaluator may pose a risk to them in their communities (e.g. interviewing women in certain communities, or young people who belong to certain gangs).
Hence, the evaluator at the evaluation planning stage must identify not only which situations in the course of the evaluation are more likely to pose a risk to evaluation participants, but also why and how, and put measures in place to avoid these. The ‘do no harm’ rule must be core to evaluators’ considerations on risks and mitigation strategies.
Confidentiality and anonymity
Participants in an evaluation should usually be guaranteed confidentiality. This means that, regardless of the information they provide, they will not be identified as the source. Additionally no statements or other type of information that may identify participants should be shared with others or publicly displayed by the evaluator. It is important to note and to make it absolutely clear to participants that statements and information provided confidentially and on condition of anonymity can nevertheless appear in the evaluators’ documents and reports with the proviso that these do not directly identify the source.
Traditionally confidentiality is afforded through anonymisation. This is often relatively easy to achieve in quantitative studies involving large numbers, but much harder to achieve in qualitative studies or studies involving small numbers of participants. For example, when an evaluator is evaluating a relatively small programme that involves a small number of stakeholders the evaluators must know that participants can be easily identified via their quotes. Hence the use of interviews as a method should be reconsidered.
The right to confidentiality is not absolute. For example, in the case of children and depending on the jurisdiction in which the evaluation takes place there may be a requirement to disclose information about a child at risk of harm that overrides any offer of confidentiality.
Guiding principles that relate to evaluators’ inherent ethics
Professional integrity
Often when it comes to professional integrity what is emphasised from an ethical point of view is the evaluators’ obligation to disclose their inability to carry out the evaluation. This can be due to a lack of the necessary qualifications, experience, or a conflict of interest.
However, it should also encompass the evaluators’ readiness to ‘[D]ecline evaluation assignments where the client is unresponsive to their expressed concerns that the evaluation methodology or procedures are likely to produce a misleading result’ (UNEG 2007:6).
This, of course, means that professional integrity obliges any evaluator to ensure that external and internal pressures do not succeed in biasing the findings and unduly influence the evaluators’ recommendations. It is the equivalent of Church’s principle of freedom from political interference whereby ‘[E]valuations should be conducted free from political interference from the implementing organization, the donor, the evaluation team, and the stakeholders’ (Church and Rogers 2006: 195).
Openness to and respect for diversity
Evaluators must respect participants’ differences in culture, religion, beliefs and practices, ethnicity, gender roles, disability and age, and be aware of the implications of these differences, reflecting on them when deciding on the methods, approaches, and reporting. For instance, certain methods should not apply to young people and children and alternatives will need to be considered (Williamson 2005), indigenous groups (Williams et al. 2011) or women victims of domestic violence (Sullivan and Cain 2004).
Additionally, evaluators must ensure that the diversity contained in the wider context is also reflected in the evaluators’ sample of participants, ‘including relatively powerless, “hidden”, or otherwise excluded groups’ (UNEG 2007: 7).
Cultural competency
More generally, cultural competence in evaluation tends to be defined as:
a systematic, responsive inquiry that is actively cognizant, understanding, and appreciative of the cultural context in which the evaluation takes place; that frames and articulates the epistemology of the evaluative endeavor; that employs culturally and contextually appropriate methodology; and that uses stakeholder-generated, interpretive means to arrive at the results and further use of the findings. (SenGupta et al. 2004: 13, bold added)
The above definition, however, does not sufficiently highlight the most vital aspect of a culturally competent evaluator, namely their ability to examine the culture-based assumptions – their own and of those being evaluated (Nelson-Barber et al. 2005: 62), understand the impact that these may have on the evaluation (namely in terms of the salient cultural issue of power asymmetries), and pro-actively address these through, for example, avoiding certain methods and approaches (Chouinard and Cousins 2007: 46).
Limitations of ethical guidelines
There are criticisms made of attempts to set out ethical guidelines and some are of a practical nature. For example:
Codes