Evaluation in Today’s World. Veronica G. Thomas
profession. By the end of the 1960s, program developers, sponsors, and evaluators recognized the need for rigorous standards to guide program evaluation. In 1974, representatives from three national professional associations joined together to form the Joint Committee on Standards for Educational Evaluation. The committee developed and disseminated three sets of standards including the Program Evaluation Standards, the Personnel Evaluation Standards, and the Student Evaluation Standards. In 1994, the membership of the AEA Board approved the Guiding Principles for Evaluators intended to be a guide to the professional ethical conduct of evaluators. (The Program Evaluation Standards and the Evaluators’ Ethical Guiding Principles, a later version of the Guiding Principles for Evaluators, were discussed in greater detail in Chapter 2.)
Methodological Approaches and Paradigm Wars
Evaluation in the 1960s and 1970s was undoubtedly a quantitative enterprise. During the 1960s and 1970s, new developments in techniques and methodologies appeared that promised to raise the overall quality of evaluations. Perhaps the most impressive and substantial achievements during this time were those of large-scale field experiments. Each of the country’s major federal departments that operated social programs had at least one large-scale field experiment taking place during the 1970s, including, for example, (a) housing allowance experiments (U.S. Department of Housing and Urban Development); (b) experiments on transitional aid to release prisoners and on supported work (U.S. Department of Labor); and (c) enhanced police patrolling (U.S. Department of Justice) (cited in Rossi & Wright, 1984).
Leaders in the field at that time, such as Donald T. Campbell, Peter H. Rossi, and Carol H. Weiss, generally argued for evaluation to take advantage of the methods of research and analysis being utilized in the most prestigious domains (mostly quantitative methods) of social science such as psychology and sociology. As stated earlier in this chapter, Suchman (1967) published one of the earliest textbooks in evaluation, Evaluative Research: Principles and Practices in Public Service and Social Action Programs, which also placed emphasis on use of the experimental design (discussed in greater detail in Chapter 11).
In evaluation, like other social and behavioral science fields, during the 1970s and 1980s the dominant use of quantitative methods came under attack as part of the paradigm wars. There was increasing recognition that while experimental design may work in small-scale evaluation studies, it is more “difficult” or “unworkable” in the context of large-scale social and educational programs, resulting, in part, in findings showing program failure (Brandon & Sam, 2014). The 1978 meeting of the Evaluation Research Society devoted substantial program time to consideration of qualitative methods (Patton, 1980). With the initial publication of Qualitative Evaluation Methods, Michael Quinn Patton (1980) provided evaluators with a reference for expanding their methodological approaches to include qualitative methods. In his book, Patton argues that
the issue of selecting methods is no longer one of the dominant paradigm versus the alternative paradigm of experimental designs with quantitative measurement versus holistic-inductive design based on qualitative measurement. The debate and competition between paradigms [are] being replaced by a new paradigm—a paradigm of choices. The paradigm of choices recognizes that different methods are appropriate for different situations. (p. 20)
By the mid-1980s, the evaluation literature was beginning to reflect the potential value of integrating quantitative and qualitative methods for the purposes of both triangulation (L. Smith & Kliene, 1986) and enhancing the rigor and credibility of evaluations (Silverman, Ricci, & Gunter, 1990).
Two Influential Scholars’ Contributions to Methodological Approaches of the 1960s–1970s
Many scholars contributed to discourse on evaluation methodologies in the 1960s and 1970s. In the sections that follow, we focus on probably the two most prominent individuals of the time: Donald T. Campbell and Lee J. Cronbach. They are among the most influential figures who first advocated different methodological views of evaluation methods in their writings.
Donald T. Campbell.
Shadish and Luellen (2004) refer to Donald T. Campbell as the “accidental evaluator” given his central importance in the field of evaluation. While Campbell did not start out intending to be an evaluator, his dedication to understanding causality behavior and how to solve social questions led him to this field. The evaluation field came to embrace Campbell’s works, including “Reforms as Experiments” (1969) and Experimental and Quasi-Experimental Designs for Research (D. Campbell & Stanley, 1963), which were probably his most influential writings on evaluation methods (Rossi et al., 2004; Shadish & Epstein, 1987). According to Campbell (1994), “Reforms as Experiments” was his first publication targeting a program evaluation agenda. Campbell and Stanley’s (1963) work on research designs created an entirely new vocabulary for the taxonomy of research designs and illuminating validity issues (Rossi & Wright, 1984). Campbell introduced issues that made their way into thinking about evaluation findings. As is covered in detail in Chapter 11, these include concepts such as internal validity, external validity, and threats to validity. Campbell’s work was instrumental in making the randomized controlled trials (or designs), with their emphasis on net effects, the design of choice in program evaluations throughout the 1970s and 1980s. It is worth noting that Campbell and Stanley did put forth the quasi-experimental design option when that was all the situation allowed, thus allowing evaluators and other researchers to test hypotheses in less-than-ideal field settings.
Numerous scholars and practitioners have challenged Campbell’s “experimenting society” approach that called for use of experimental methods in evaluation. The major points of contention and conclusions, as summarized by Shadish and Luellen (2004), are that
experimental methods were insufficient to address social problems in a world where policy practice is entangled with politics, economy, and social pressures; questioned the importance of noncausal questions and nonexperimental methods; complained that experimentally based knowledge was not fully implemented in solving social problems; and pointed out limitations of experimental methods. Eventually, the field of evaluation rejected Campbell’s Experimenting Society as too narrow and Utopian, preferring a broader vision of the role of evaluation. Even so, because bias remains a central problem for evaluation, the solutions Campbell offered will be his greatest legacy. (pp. 82–83)
Lee J. Cronbach
The work of Lee J. Cronbach also was influential in evaluation and a contrast to the approach of Donald T. Campbell. As early as 1963, Cronbach challenged the evaluation community and pointed out that the current level of evaluation practice was wholly inadequate to meet the needs of the newly developed, federally sponsored curriculum reforms (O’Sullivan, 2004). He was skeptical of the view of evaluation as sterile, detached, objective scientific activities.
Cronbach did not champion a particular methodology for evaluation, and he valued methodological pluralism in evaluation centered on “better understanding society’s enduring social problems and the ways in which the specific policy and program being evaluated are meaningfully addressing one such problem” (Greene, 2004, p. 172). Cronbach’s work emphasized the limitations of randomized field trials, the importance of local contexts on performance, and the social and political aspects of program evaluation. As such, he focused not on the technical aspect of measurement in evaluation, but on the policy-oriented nature of evaluation. Cronbach’s evaluation research influenced program evaluations across various fields such as health and criminal justice reform, from health programs to juvenile delinquency programs. Cronbach coauthored two influential books in the evaluation field: Toward Reform of Program Evaluation (1980), which was written by a team of consortium faculty led by Cronbach, and a parallel volume, Designing Evaluations of Educational and Social Programs (1982).