Evaluation in Today’s World. Veronica G. Thomas
of something, or the product of that process” (p. 139). Several years earlier, Yvonna Lincoln and Egon Guba (1985) defined evaluation as “disciplined inquiry” with a goal of determining value for program improvement or refinement. Their definition also made a distinction between formative evaluation (to determine the value of a project, program, or product in order to improve or refine it) and summative evaluation (to determine the worth, value, and/or success of a project, program, or product), both of which are discussed in further detail in Chapter 6. In 1997, Michael Quinn Patton went a step further in his definition, adding that the information collected in evaluation could be used to inform decisions about future programming as well as “to make judgments about the program [and] improve program effectiveness, and/or inform decisions about future programming” (p. 23). Trochim’s (1998) definition includes providing evidence in decision making and contexts as well. It also describes those contexts as “inherently political” and as involving “multiple and often conflicting stakeholders, where resources are seldom sufficient and where time pressures are salient” (p. 248).
More recent definitions (e.g., Newcomer, Hatry, & Wholey, 2015; Rossi, Lipsey, & Henry, 2019) refer explicitly to social science methods, with Rossi et al. (2019) including in their definition a reference to context and an explicit goal to “inform social action to improve social conditions” (p. 6). While myriad descriptions of evaluation are found in the literature, consistent across them is the idea that evaluation is a systematic, applied inquiry process for collecting and synthesizing evidence (data) and drawing conclusions about the state of affairs, value, merit, worth, significance, or quality of an entity.
The focus on value is key to understanding what evaluation is. Value is the “feature that distinguishes evaluation from other types of inquiry, such as basic science research, clinical epidemiology, investigative journalism, or public polling” (Fournier, 2005, p. 140). Unlike research, evaluations do not simply report outcomes; they draw conclusions about the value or quality of those outcomes within a particular context and for specific groups. Program evaluation, in particular, involves the use of research methods to examine a program’s goals, objectives, outcomes, and impact. It can also be used to investigate a program’s structure, characteristics, activities, organization, and political and social environment. Evaluation has the potential to enable society to meaningfully learn about its persistent social problems and how to effectively solve them (Cronbach et al., 1980).
Our definition of evaluation encompasses many of the components in the earlier definitions. It includes systematic inquiry, assessing value and awareness of context, and also ethical, quality, justice, and cultural concerns. We define evaluation as a
disciplined inquiry involving the systematic, contextually responsive, and ethical application of research tools and methods to collect data that assess the effectiveness and operations of programs within the various social, political, and cultural contexts in which they operate. Evaluation’s ultimate goal is to provide credible evidence that fosters greater understanding and improves decision making, all aimed at improving social conditions and promoting healthy, just, and equitable communities.
Evaluation Characteristics
Evaluation is not simply a scientific endeavor in search of “truth” and “solutions.” Evaluation, while complex, is often less concerned with general truths and generalizations because it focuses on specific programs and practices taking place within a specific context. This makes evaluation much more an idiosyncratic activity that must be tailored to the particular circumstances under consideration. “Evaluation is not an examination into the inert, static, and external realities of programs but instead, into the fluid subjective world of people’s lives as experienced, interpreted, recalled, and mediated by them and the, oftentimes, racialized contexts of the systems that programs, communities, and individuals are embedded” (Thomas et al., 2018, p. 156). The complexity of social programs makes it critical that anyone who is tasked with evaluating such programs understand the context of the program and the evaluation. Chapter 9 covers this aspect in greater detail.
Evaluation is very much a social enterprise that is best understood by taking into consideration the social, cultural, economic, and political contexts surrounding the program under consideration. Attention to public interest and public good is a critical aspect of the evaluation process. Evaluators cannot ignore the reality that they become a part of the never-ending struggle to make judgment calls about social activities that create the conditions or obstacles for social mobility (Waters, 1998). In 1980, Cronbach et al. pointed out that program evaluation was a process by which society learns about itself. Melvin Hall (2018a) used that point to underscore the need for evaluators to take up more space in the public sphere where institutionalized sources of potential racism and classism should be identified and interrogated. His call was for evaluative thinking, as discussed in the next section, to become more prominent in public debate and policy reviews—an appeal to evaluators to identify and engage the important societal issues embedded in the work we do. Hall’s charge is an integral part of the underlying thinking of this book. The social justice focus of this book is not just limited to race and class but includes other social justice issues such as disability, sex or gender, sexual orientation, and socioeconomic status. In the activity that follows, readers begin to develop their own definitions of evaluation.
Activity: Defining Evaluation
In small groups, discuss the characteristics you think should be included in a definition of evaluation.
Evaluative Thinking
The concept of evaluative thinking is an increasingly important topic in evaluation and a key component of evaluation capacity and practice (e.g., Baker & Bruner, 2012; Patton, 2008), However as Buckley, Archibald, Hargraves, and Trochim (2015) point out, definitions of evaluative thinking are varied and sometimes ambiguous. They hold that evaluative thinking is, “in essence, critical thinking applied to contexts of evaluation” (p. 376). Other definitions of evaluative thinking describe it as a type of reflective practice, which is “a way of studying your own experiences to improve the way you work” (Brightside, 2020, para. 1). Baker and Bruner (2012, p. 1) see evaluative thinking as a reflective practice that “fully integrates systematic questioning, data, and action into an organization’s work practices” while Michael Quinn Patton (quoted in Waldick, 2011, para. 13) describes it as “an analytical way of thinking that infuses everything that goes on.” After extensive review of the evaluation thinking literature, Buckley et al. (2015, p. 378) proposed the following definition: “Evaluative thinking is critical thinking applied in the context of evaluation, motivated by an attitude of inquisitiveness and a belief in the value of evidence, that involves identifying assumptions, posing thoughtful questions, pursuing deeper understanding through reflection and perspective taking, and informing decisions in preparation for action.”
Reflective practice can be an important component of evaluative thinking. Reflection and reflective practice can catalyze evaluators to
collect information before making up one’s mind;
seek various points of view before coming to a conclusion;
think extensively about a problem before responding;
calibrate the degree of strength of one’s opinion to the degrees of evidence available;
think about future consequences before taking action;
explicitly weigh pluses and minuses of situations before making a decision; [and]
seek nuance and avoid absolutism. (Stanovich, 2010, p. 36)
Evaluative thinking can start by simply asking some questions and investing in the process of answering them. Sometimes, the reflection and discussion themselves are as important as any answer you might come up with (IllumiLab, 2018a). The following activity provides some questions that readers can ask to help promote