Evaluation in Today’s World. Veronica G. Thomas
Black) might be pertinent for an evaluation of a higher education program while the culture of correctional faculty (i.e., maximum, medium, or minimum security) might be pertinent for an evaluation of in-house recidivism programs. Other cultural groups might be important in different evaluations. However, it is important to remember that “accurate predictions of individual behavior based on nationality or other collective-level categories are typically not possible and may even cause offense. We are all members of multiple different groups and these identities become relevant at varying moments of our day” (Handford et al., 2019, p. 45).
As covered further in Chapter 13, how participants in programs and projects being evaluated identify themselves or how they are identified by others is one way to help evaluators understand cultural group memberships that may be salient for the evaluation. Learning about what is currently going on in relevant institutions and the surrounding communities can also help to identify other salient factors. Evaluators need to have ongoing knowledge and understanding of the cultural groups and related factors.
As covered in greater detail in Chapter 5, evaluators need to be culturally competent, and their evaluations need to be culturally responsive. SenGupta, Hopson, and Thompson-Robinson (2004, p. 13) go beyond the relatively simple definition of cultural competence given in the beginning of this chapter to describe cultural competence in evaluation as a “systematic, responsive inquiry that is actively cognizant, understanding, and appreciative of the cultural context in which the evaluation takes place; that frames and articulates the epistemology of the evaluative endeavor; that employs culturally and contextually appropriate methodology; and that uses stakeholder-generated, interpretive means to arrive at the results and further use of the findings.” Cultural competence in evaluation takes place through a continuing open-ended series of substantive, ethical, and methodological insertions and adaptations that aligns the inquiry process with the characteristics of the groups/contexts being examined.
It is not enough for evaluators to embrace cultural competence, but as covered in detail in Chapter 5, evaluators must mindfully apply that competence to every aspect of their work from evaluation planning to reporting and use of results. Additionally, cultural competence is important in evaluation scholarship and in evaluation educational and professional development settings.
One of those areas where this was evident for us was in determining the most culturally appropriate terminology to use throughout this book to describe people from different races/ethnicities. There are many different ways to do this, and there is no one right answer. It is particularly difficult because of the practical necessity of using one term to collectively describe diverse subgroups. Our choice has been to acknowledge the complexity and be transparent about the choices we made. Since Asian American is the common term for people in the United States of Asian descent, we chose to use that term, knowing that it doesn’t acknowledge the great diversity within that group. We were unsure as to whether we should use American Indian or Native American to collectively identify members of the 562 different tribes in the United States. We asked three people from different tribes who are very active in their tribal communities for advice. While they have used American Indian in the past, their preference is for Indigenous People, which is the term we are using.
We chose to use the terms African American and Black fairly interchangeably; however, we tend to use African American when we refer to people whose origins are in the African continent but whose history is on the American continent and Black when we are speaking more generally. We also chose to use Hispanic as a generic collective term because it encompasses people from outside as well as inside Latin America.
The Impact of Politics
In fields like evaluation, where results are used not just for program improvement but for making policy decisions and determining funding, the climate in which the evaluation is done is often challenging and politically driven. In a time when there is increasing distrust of research and evaluation, doing evaluations becomes increasingly difficult. While in recent years the challenges have seemed to be greater, there has always been tension between science, including evaluation, and politics. In 2008, Chelimsky explained that since “our government’s need for evaluation arises from its checks and balances structure, evaluations working within that structure must deal not exceptionally, but routinely and regularly, with political infringement on their independence that result directly from that structure” (p. 400). She went on to point out the irony that “what should surprise us would be the absence of pressure on evaluators to make an agency ‘look good’ or the lack of effort by agency managers to try to manipulate the work of evaluators implementing legislative oversight” (p. 400).
Another way that politics can intertwine with evaluation is the political interest in “quick fixes” and the unwillingness to acknowledge underlying factors. As Thomas et al. (2018, p. 517) point out,
Evaluators are often asked to assess the effectiveness of social programs that are designed to yield a quick “magic bullet” to fix to problems derived from years of racial oppression. Here, race is and is not the problem inasmuch as racism fuels the disparities we witness. However, this reality is virtually absent in the discourse of numerous commentaries and policy makers who are quick to cite Black failure or pathology without examining the historical root causes. As a result, social programs seek to address outcomes, such as the achievement gap and health disparities, rather than the race-based structural inequalities in the social, economic, and political systems that contribute to these outcomes.
As the following case studies indicate, evaluation results can and at times do lose out to politics.
Case Studies: Evaluation Results vs. Politicians
21st Century Community Learning Centers
An extensive evaluation of the 21st Century Community Learning Centers, a federally funded after-school program, found that the program did not affect student outcomes. Those findings were met with much resistance from those who strongly advocated for the program. For example, then California governor Arnold Schwarzenegger used strong community support and “anecdotal evidence” to justify his support for the program, which continues to exist. Describing the response to the evaluation results, Ron Haskins pointed out “any sentence akin to saying ‘Everybody knows this program works’ is an enemy of evidence-based policy” ( J. White, 2017, p. 11).
Project DARE (Drug Abuse Resistance Education)
Project DARE is one of the most widely used substance abuse prevention programs targeted at school-aged youths. It has been the country’s largest single school-based prevention program. Multiple evaluations and meta-analyses have found DARE to be ineffective, yet it continues to be widely used, paid for by federal funds (West & O’Neal, 2004). In 2017, then attorney general Jeff Sessions supported DARE because he “firmly believed” in its effectiveness, regardless of what the data said. Knowing that the program has been found to be ineffective, some school districts continue to use it for a variety of reasons including their belief that no short-term program can change students’ drug-taking behavior and that, if it improves student/police relationships, that is enough to keep it going (Ingraham, 2017).
While some of the reasoning behind these examples is political, there can also be what Tom Kibler (personal communication, first quoted in Campbell, Hoey, & Perlman, 2001, p. 34) described as the “pure of heart model” influencing the decisions. The model is based on the premise that “since my heart is pure and my cause is just and I work really hard at it, the change I am seeking will happen.” The pure of heart model is often held by caring people who are trying to help others who feel strongly what they are doing is right. If the data don’t support that, then the data are wrong. As one program leader explained, “We are doing this on faith and if you don’t believe in it, **** you” (Campbell et al., 2001, p. 35).
There can be issues with doing things on faith. Even