Evaluation in Today’s World. Veronica G. Thomas
the Role of Evaluation
The proliferation of ambitious new social programs beginning with the Great Society initiative in the 1960s begged the important questions “Does this program work?” and “Does this program work better than something or nothing?” Many believed that requiring evaluation of initiatives would help build a body of knowledge regarding which policy ideas and programs are effective. The Perry Preschool Project, a high-quality preschool program for African American children from disadvantaged backgrounds carried out from 1962 to 1967, has been characterized as a social program that worked based on evidence from experimental evaluation design. The following activity invites students to reflect on and discuss an evaluation of the Perry Preschool Project.
Reflect and Discuss: The Perry Preschool Project
Read the following description of an evaluation of the Perry Preschool Project. Then answer the questions that follow.
Program Description: A high-quality preschool program for children from disadvantaged backgrounds, the Perry Preschool Project was conducted from 1962 to 1967 but led to a longitudinal study.
Evaluation Purpose: The Perry Preschool Project evaluation sought to determine whether access to high-quality education could have a positive impact on preschool children and the communities where they live.
Methodology: The Perry Preschool Project was a small, but well-conducted, randomized controlled trial with a sample of 128 three- and four-year-old African American children living in poverty and assessed to be at high risk of school failure. The preschool children were randomly divided into two groups: One group entered a high-quality preschool program, and a comparison group received no preschool education. Investigators continued to follow the Perry Preschool Project participants throughout their lives in this landmark study.
Selected Key Findings: Evaluation results yielded large effects on educational attainment, income, criminal activity, and other important life outcomes, sustained well into adulthood.
At age 27, the participants who experienced the preschool program
completed an average of almost 1 full year more of schooling (11.9 years vs. 11 years);
spent an average of 1.3 fewer years in special education services—for example, for mental, emotional, speech, or learning impairment (3.9 years vs. 5.2 years); and
had a 44% higher high school graduation rate (65% vs. 45%).
At age 40, the participants who experienced the preschool program
had fewer teenage pregnancies;
were more likely to have graduated from high school;
were more likely to hold a job and have higher earnings;
had committed fewer crimes; and
were more likely to own their own home and car.
Source: Adapted from https://highscope.org/perry-preschool-project/
Brainstorming Questions
1 What social, political, economic, and racial issues were happening in the country that might have influenced the conceptualization, design, and outcome of the evaluation?
2 If you were the evaluator of this project in 1969, what would be the top two historical factors that you would consider during your work? Why?
Unfortunately, evaluation findings from many other high-profile social programs yielded dismal measurable results. This resulted in some rethinking of evaluation and its purpose, methodologies, and problems. Edwards, Guttentag, and Snapper (1975) offer what they refer to as five complaints about folkways of evaluation research that were the prevailing sentiment of the time and resulted in more rethinking of evaluation’s purpose and methodology. These include reification of programs; insistence on causal inference; conducting pseudo-experiments; planning the formative, then summative, evaluation in a stage-by-stage sequence; and a baseball statistical approach to handling lots of data. Table 3.3 elaborates on their complaints.
Despite some successes, one of the most important lessons to be learned from all the evaluations initiated during the 1960s and 1970s was the difficulty in carrying out randomized controlled experimental designs in field settings. Due, in large part, to the dismal findings from large-scale randomized controlled trial evaluation studies of the late 1960s through the end of the 1970s, many evaluators became disheartened with the traditional approaches to evaluation. In fact, a reasonable summary of the large-scale field experimental evaluation findings of the 1960s and 1970s is that the expected value of the effect of any program hovered around zero (Rossi & Wright, 1984). Of course, such a conclusion was quite disheartening to the social reformers who had hoped that the Great Society programs would result in significant improvements in the lives of poor people and marginalized communities. Given this reality, evaluators had to contend with two possibilities: (a) that the implementation of social programs of the day truly had little to no impact on the intended outcomes for their participants, or (b) that the existing evaluation models and methods being used were inadequate to capture the outcomes of interest.
Table 3.3
Source: Edwards, Guttentag, & Snapper (1975).
With the evaluation challenges of the 1970s and 1980s, there were increasing concerns among evaluators, particularly around how evaluation was viewed and its methodology. L. Ross and Cronbach (1976), for example, emphasized moving away from the mainstream view of evaluation that characterizes evaluation as an event that begins, runs alongside a program for a time as the evaluator makes observations and collects data, and ends rather abruptly with a report to an all-powerful decision maker (usually someone outside the program under consideration). In this mainstream view, the evaluator, essentially, enters the picture only after the initial events in the life of the program have begun. Instead, they argued for a better approach to evaluation, whereby evaluation becomes a component of the evolving program itself rather than simply providing “disinterested monitoring undertaken to provide ammunition to the warring factions in a political struggle” (L. Ross & Cronbach, 1976, pp. 18–19).
Influential Women in Evaluation: 1970s–1990s
Since the 1970s, numerous (mostly white) women have contributed to evaluation scholarship and practice. Based on our review of the AEA’s listing of past presidents, from its founding in 1986 through 2020, the association, unlike many other professional organizations in this country, had about equal proportion of men and women presidents (i.e., 17 female presidents and 18 male presidents). So, women certainly have been represented in the leadership of the discipline’s major professional association.
In the section that follows, we focus on some of the women that made significant contributions in evaluation between the 1970s and the early 1990s. In particular, we highlight the contributions of seven women who contributed directly to evaluation theory, method, policy, and/or practice through their work in academia, government, and evaluation practice communities. We highlight Carol H. Weiss, Yvonna S. Lincoln, Eleanor Chelimsky, Lois-Ellin Datta, Floraline I. Stevens, Laura Leviton, and Beatriz Chu Clewell. Similar to the earlier section on “hidden figures,” we also included a photo of each woman featured in this section.
Carol H. Weiss
Carol H. Weiss was probably the most prominent female evaluation theorist and practitioner of her time. In the mid-1960s, Weiss evaluated a Harlem-based training project as part of President Johnson’s War on Poverty. This yielded invaluable lessons that Weiss later disseminated through her scholarship. Her 1972 book, Evaluation Research: Methods for Assessing Program Effectiveness, provided an in-depth analysis