An Introduction to Evaluation. Chris Fox
they hope to, the ‘messy’ reality of the field, the unanticipated effects, conflicting political priorities and all the personal factors –fears, anxieties and differences involved in the socio/political engagements of evaluation on the ground … (Simons 2006: 250–1)
It has also been pointed out that codes tend to concentrate on avoiding harm, rather than promoting good (ibid.). However, in this section we concentrate on two more fundamental criticisms. The first is the accusation that commonly accepted guidelines for ethical practice are ‘Western-centric’, and the second is that guidelines are designed primarily to protect the interests of the evaluation profession.
The ‘anthropo’-centric approach to ethics in evaluations
There are nevertheless critiques to the main and commonly accepted evaluation guiding principles. Schwandt very directly writes that most principles ‘have been developed largely against the foreground of a Western framework of moral understandings’ (2007: 400). More specifically, Schwandt argues that these principles:
tend to regard, without question, notions such as individualistic self-expression, personal autonomy, individual growth and goal attainment, egalitarianism, meritocracy, and competitiveness, as normative. (2007: 400)
In short, the main guiding principles that are the most commonly advised by different professional organisations may not be recognised and understood in all cultures. Schwandt finds it relevant to introduce this element of cultural relativism, emphasising the point that ‘the meaning of acting ethically (being just, fair, honest, etc.) is shaped, in a significant way, by how persons are situated in cultural contexts’ (2007: 401).
There may indeed be several contexts where this cultural relativism may require a more flexible approach. For instance, in certain cultures granting women confidentiality and anonymity by having individual, semi-structured interviews with them this may lead to retaliation from their husbands/partners, who may feel uncomfortable with their wives being interviewed privately. In these cases, the evaluator must allow the principle of ‘do no harm’ to prevail over the principle of confidentiality and anonymity, and either talk to women in their family context or in the context of a focus group discussion.
Protecting the profession
A criticism sometimes levelled at ethical codes is that they are self-serving in nature, being designed to protect the interests of the profession rather than the researched (Simons 2006). Similar accusations have been made about the role of ethics boards (see above) which might, in some cases, be seen as more concerned with defending reputations and preventing litigation rather than being led purely by ethical concerns.
Responding to the criticisms of ethical guidelines: the evaluator as a self-reflective negotiator
In sum, and following on from this element of cultural relativism applied to evaluation ethics, the major quality required in an evaluator is their Verstehen capacity. This stands for the evaluator’s ability to empathise with the context, be self-reflective regarding their own positions (ethical and others) within that specific context, and be able to negotiate around common understandings and differences. This resonates strongly with cultural competency where the evaluator’s capacity to be reflexive is paramount. As Sultana (2007) notes that:
being reflexive about one’s own positionality is [not] to self-indulge but to reflect on how one is inserted in grids of power relations and how that influences methods, interpretations, and knowledge production. It is also implicated in how one relates to research participants and what can/cannot be done vis-à-vis the research within the context of institutional, social, and political realities. (Sultana 2007: 376)
Hence, and whilst the ethical guiding principles are useful, these must be seen mostly as pointers rather than rules that have to be applied regardless. Empathy and the ability to negotiate ethically the ethics that apply to the context are the most important.
The challenges that evaluators must overcome to ensure ethical evaluations
In this section, two main types of challenges are mentioned: those relating to the implementation of certain methods, and those relating to a number of exogenous factors (i.e. factors outside the evaluator’s control).
Methods and ethics
Methods, and their implementation, may be one of the main sources of concern when it comes to ethics in evaluations. Some methods however have proven more contentious than others, and more often than not these sit at opposite ends of the quantitative/qualitative continuum: randomised control trials and participatory action research (PAR). These are used below to illustrate how certain methods can raise issues for the ethics of evaluations.
Randomised control trials
Randomised control trials as a methodological approach have their origins in clinical research. The use of this approach in social programmes gained traction in the early 2000s and garnered fierce proponents and ferocious critics. The balkanised debate went public via some specialised journals (e.g. Journal of Multidisciplinary Evaluation) and some umbrella evaluation associations felt the need to publicly express their position on this (Barahona 2010; Barrett and Carter 2010; Oakley et al. 2008).
In September 2007, the European Evaluation Society (EES) published a statement advocating for the use of diverse methodological approaches to evaluation, noting that the ‘EES however deplores one perspective currently being strongly advocated: that the best or only rigorous or scientific way of doing so is through randomised controls trials (RCTs)’ (European Evaluation Society 2007: 1).
Nevertheless, largely absent from this debate were some important ethical considerations associated with the use of the approach in non-clinical settings, some of which may pose considerable challenges.
Barahona (2010) contrasts informed consent in clinical research trials with informed consent in the context of impact evaluations on social development programmes:
When RCTs are used in clinical research, during the process of recruiting subjects these subjects are informed about a range of factors, such as the trial objectives, the treatments, the risks, the random nature of treatment allocation, the support available to participants, the mechanisms for opting out, and the procedures for providing comments and submitting complaints. This process concludes with each subject agreeing to be included in the trial. When RCTs are proposed for impact evaluation, the issue of consent from participants is not discussed. (Barahona 2010: 16)
And even when the issue of informed consent is discussed, Barrett and Carter (2010) argue that it is so often ‘at the ethically-questionable cost of sacrificing the well-accepted right of each individual participant to informed consent, as well as the corresponding obligation of the researcher to secure such consent’ (Barrett and Carter 2010: 520); this at least is the case for cluster-randomised trials. Indeed evidence of individual informed consent being requested by impact evaluations that use communities as control groups is scarce.
Additionally, Barrett and Carter (2010) and also Scriven (2010) raise the relevant issue of the (double) blindness un-blindness of the RCTs’ subjects/researchers. Commonly in clinical trials neither participants nor researchers assigned to the groups are aware of who is being administered the placebo or the drug. This is in part to avoid participants’ emotional distress. In social programmes’ contexts, participants will be aware of the fact that they are not having the access to intervention that the other participants are.
Finally, there is the very real possibility of violating the ethical guiding principle of ‘do no harm’. Barrett and Carter (2010) provide some examples of impact evaluations in the context of which control groups were incentivised to engage in actions that were dangerous to themselves and others.
As discussed in Chapter 12, the counter argument made by supporters of experiments is that