An Introduction to Evaluation. Chris Fox
should not be given to people, and a randomised field experiment, because of its high internal validity, is the best evaluation design for showing that an intervention is effective. In support of this counter argument Chalmers highlights what he sees as the double standard whereby many rigorous ethical processes are in place to regulate when and how RCTs are conducted, and yet practitioners routinely use untested interventions without having to meet the same ethical standards:
Professionals who are uncertain about whether a particular intervention (a policy or practice) will do more good than harm, and so wish to offer it only within the context of a controlled trial so that they protect people in the face of current uncertainty and learn about its effects, are expected to observe elaborate informed consent rituals. If exactly the same intervention is offered by other professionals because it was recommended during their professional training three decades previously, or because there is a plausible theory that suggests it will be helpful, or because it is an accepted routine, or because they or the institutions for which they work have a vested financial or political interest in promulgating it ... the standard of consent is relaxed. (Chalmers 2003: 31)
Participatory action research (PAR)
Smith et al. (2010) emphasise that:
[I]n PAR, professional researchers do not enter communities to conduct studies on community members. Rather, they form partnerships with community members to identify issues of local importance, develop ways of studying them, collect and interpret data, and take action on the resulting knowledge. (Smith et al. 2010: 407–8)
Ethical issues arise from this total immersion of the researcher/evaluator in the context which they are researching/evaluating. Again, the most obvious one here is informed consent. As in the case of cluster-randomised trials, participatory action research involves working with a group of individuals or communities. The challenge is greater in the case of PAR. Cluster-randomised trials work with a fixed number of research subjects that remain the same throughout the evaluation. In PAR, research subjects with whom the evaluator interacts may change. This poses obvious challenges to the ethical principle of voluntary participation; members of the group or community are not so aware that they are part in a study. This reflection is not at the front of their minds when interacting with the evaluator. Perhaps in many cases if this was the case, they might have decided not to volunteer information.
The evaluator’s immersion in many cases demands a more flexible approach to information gathering that is not confined to private exchanges as in the context of semi-structured interviews. Anonymity and confidentiality may also be at stake and there are other ethical-methodological issues that may also arise:
including defining what counts as a ‘community’; the potential for conflict between individual and group interests; how to modify informed consent to take account of group characteristics; and issues of who best represents a group or community. (Durham Community Research Team 2011: 8)
Finally, issues may arise with regard to power asymmetry and the ‘blurring of the boundaries’ in PAR. PAR proponents often mention evaluators’ ability to best apprehend power relations by being immersed in the group or community. Nevertheless, evaluators may be unable to see themselves as part of this configuration and understand both their bias and how they are seen by others (i.e. others’ bias towards them). Additionally, evaluators may struggle to remain impartial and thus side with one side of the argument, thereby blurring the boundaries between evaluator/member of the group/community (Durham Community Research Team 2011).
From the above, it seems likely that all ethical guiding principles are sensitive to methods’ implementation. Moreover, regardless of where a method sits on the quantitative/qualitative continuum ethical challenges seem to be rather common. The issue is therefore to identify what can be done in order to ensure the integrity of the principles.
Exogenous factors and ethics
There are a number of factors outside evaluators’ control that may pose considerable ethical challenges. The most common ones are the literacy level of the evaluation participants, power relations, and intercultural communication.
Literacy level
Often it is thought that this issue only applies to ‘development’ evaluations, normally carried out in contexts where the (formal education) literacy levels of the evaluation participants are lower. However, ‘literacy’ must take on a broader sense to stand for participants’ mastery of the subject of the evaluation, and this can be particularly acute in the context of clinical studies or health-related evaluations. Tamariz et al. (2013) refer to health literacy, stating that:
A potential problem is that health literacy goes beyond just literacy to encompass an understanding of health-related processes, including the pathophysiology of disease and how to navigate the health system. Therefore, a patient or research subject could theoretically ‘teach back’ risks and benefits, but might not have good insight into the long-term implications of those risks and benefits. (Tamariz et al. 2013: 125)
It is very difficult for participants to consent to being part of an evaluation when their understanding of the issue is very limited. Likewise they can be quick in agreeing to participate without fully grasping what is involved (Kinnersley et al. 2011; Montalvo and Larson 2014). This is therefore one of the times where participants can only rely on evaluators’ intrinsic ethics, which will prevent them from taking advantage of participants in such situations and instead require them to invest sufficient time in explaining the main focus of the evaluation.
Power relations
Mentioned several times before, power relations and their particular configuration in a context can greatly affect the content of the evaluation as well as the ethics. Very few evaluators reflect on the fact that perhaps most participants in one way or another may be rooting for a specific evaluation outcome.
This goes beyond different groups having different opinions. It is about different groups and individuals (including those who commissioned and are paying for the evaluation) trying to influence the outcome of the evaluation by the way they behave, respond to questions, and volunteer (what type of) information. It is not about the evaluator being self-reflective and negotiating difference. It is about the evaluators being aware of the attempts at influencing and to some extent at manipulating them and reacting to these in order to avoid unethical biases.
Intercultural communication
Intercultural communication may pose considerable challenges for any evaluation. This relates to cultural relativism and the evaluator’s capacity to be self-reflective and understand their own positionality (Sultana 2007) but not only this. Evaluators can, to a large extent, improve their understanding of the cultural context they find themselves in. However others in that context will obviously not do the same. The way they communicate with the evaluator, their behaviours and attitudes, will not be adapted to the evaluator’s own cultural background. Hence this process of intercultural communication will always be needing constant ‘translation’ and interpretation in order to ensure a high degree of clarity and transparency: in the intentions of the evaluator, the objectives of the evaluation, and the information provided by the evaluation’s stakeholders.
Conclusion
Often brushed aside, ethics in evaluation are an important element. They afford transparency and integrity to the process and also enhance the effectiveness of the evaluation. It goes without saying that if ethical guidelines are duly followed, evaluators’ stakeholders will be more confident in their participation and therefore will have a higher quality input.
There is still a long way to go in this regard, and much also depends on those who commission evaluations. Commissioners should insist on stringent ethical requirements, which in turn would ensure that evaluators know and follow ethical procedures and reflect on the ethical implications of all the methods they propose to use. Evaluators should familiarise themselves with relevant ethical guidelines