Evaluation in Today’s World. Veronica G. Thomas
evaluators and public managers. While many of the criticisms of the time focused on methodological inadequacies, Weiss (1986) also pointed to issues related to the lack of fit between evaluation and the sociopolitical context of the program world. She noted that
evaluation is narrow because it focuses on only a small set of questions of importance to program people; unrealistic because it measures the success of programs against unreasonably high standards; irrelevant because it provides answers to questions that no one really cares about; unfair because it is responsive to the concerns of influential people, such as bureaucratic sponsors, but blind to the wants of others lower in the hierarchy, such as front-line staff and clients; and unused in councils of action where discussions are made. (p. 149)
Weiss (1993) called for a reconceptualization of the purpose of evaluation from simply judging program merit and worth, to also generating reliable knowledge that could guide improvement. She also stressed the need for evaluators to gain a better understanding of the political context and its influence on their work:
Photo courtesy of Martha Stewart (photographer)
Now I have more understanding how difficult it is to bring about improvement in long-standing stubborn problems, like poverty and violence. The experience of the 1960s and 1970s showed us that even under relatively favorable conditions, progress was slow and uneven. To change social conditions in the swift, massive fashion that we yearned for was much harder than we expected. I have come to have more respect for the incremental changes that evaluation helps to bring about. If evaluation contributes to making small continuing improvements in current policy, as I think it does, this is no small beans. It is well worth our time and effort. (Weiss, 1993, p. 109)
Probably Weiss’s major contribution to the historical evolution of evaluation is her sustained effort to push the field toward better recognition of programs as not neutral, antiseptic, laboratory-type entities but instead as entities that emerge from the “rough and tumble” of political support, opposition, and bargaining—attached to which are the reputations of legislative persons, the careers of administrators, the jobs of program staff, and the expectations of clients. Even rigorously documented evidence of outcomes, Weiss (1993) adds, may not outweigh all other interests and concerns, and only with sensitivity to the politics of evaluation research can evaluators be as creative and strategic as they can be. Insights from Weiss’s 11 published books and more than 100 articles continue to shape how evaluators think about theory and practice evaluation.
Yvonna S. Lincoln
Yvonna S. Lincoln, the first female president of the AEA, is probably best known in social science research, more generally, and evaluation, in particular, for her contribution to qualitative methodology. Lincoln, often in collaboration with her husband, Egon Guba, championed a constructivist, qualitative approach to understanding human phenomena. Their publications, Naturalistic Inquiry (Guba & Lincoln, 1985) and Fourth Generation Evaluation (Guba & Lincoln, 1989), especially the latter book, were written, in part, to address what they believe to be the inadequacies of previous evaluation methodologies. In Fourth Generation Evaluation, the authors point out that it was their intention to define an emergent, but mature, approach to evaluation that goes beyond mere science, or just getting the facts, to include the myriad human, political, social, cultural, and contextual elements involved (Guba & Lincoln, 1989, p. 8).
A major contribution that Lincoln made to the evaluation field during the 1980s and 1990s was to highlight the limitation of the field in its focus primarily on methods (i.e., how we come to know something) and rigor (i.e., how much trust we have in what we know). In her 1990 AEA presidential address, Lincoln (1991) focused on aspects of science that the profession had failed to notice, including the science of locating interested stakeholders; the science of getting information—good, usable information—to those same stakeholders; the science of teaching various stakeholder groups how to use information to empower themselves so they can participate more fully in democratic life and in decision making; and the science of communicating results. She moved the field beyond thinking only of the “sciences of evaluation” (i.e., methods and rigor) to also thinking more seriously about the “arts of evaluation” (summarized in Lincoln, 1991, pp. 4–6). Aspects of the arts of evaluation that Lincoln focused on in much of her work, and something she urged the evaluation community to consider, include
judgment—for evaluators to be able not only to render their own judgments in such a way that they can back them up, but also to elicit the judgments of stakeholders such that both evaluators and stakeholders are clear about the values, belief systems, and community mores undergirding those judgments;
appreciation—cultivating the art of appreciating in both evaluators and stakeholders and comprehending meaning within context, understanding the social and cultural milieu from which a program draws its particular expression, and seeing something fully and in its wholeness;
cultural analysis—paying attention to rituals, symbols, and meanings that coalesce when groups of people are engaged in a common pursuit;
“hearing secret harmonies”—or learning to listen for the meanings and not just searching for the one-to-one correspondence between objectives and their achievement; and
dealing with people very different from ourselves—for evaluators to get out on the front lines more, to rely on stakeholders who want to speak for themselves, to give voice to those who cannot be heard, and to see those who have been invisible.
Lincoln is the author or coauthor of more than 100 chapters and journal articles on aspects of higher education or qualitative research methods and methodologies. Her contributions to the field of evaluation continue and have provided insight for some more contemporary approaches such as transformative evaluation (see Chapter 5) and culturally responsive evaluation.
Eleanor Chelimsky
Eleanor Chelimsky has been described as a guiding light in the field of evaluation since its inception as a profession, with several decades of experience conducting evaluations of government policies and programs, developing methodologies to respond to complex questions meaningfully, and transferring her knowledge to the evaluation community at large both domestically and internationally (Oral History Project Team, 2009). From 1966 to 1970, she was an economic analyst for the U.S. Mission to NATO, charged with statistical, demographic, and cost-benefit studies. Subsequently, from 1970 to 1980, she worked at MITRE Corporation, where she directed work in evaluation planning and policy analysis, criminal justice, and research management. Between 1980 and 1994, Chelimsky directed the Program Evaluation and Methodology Division of the U.S. GAO whose mission was to serve Congress through evaluations of government policies and programs and through the development and demonstration of methods for evaluating those policies. Chelimsky’s influence on evaluation, particularly at the federal level, has been enormous. In her position at the GAO, she pioneered the use of meta-analysis as a tool for providing program evaluation and other legislatively significant advice to Congress (Chelimsky, 1994). Chelimsky received many awards for her work, including the GAO’s top honor, “the Comptroller General’s Award,” for contributions in developing innovative approaches to evaluate the effects of government programs and fostering their use by the GAO and other decision makers in the United States and abroad.
Chelimsky has been a leader in the evaluation profession serving as president of the Evaluation Research Society in 1980 and of the AEA in 1995. She has been credited with producing nearly 300 evaluations of government policies and programs for Congress, as well as developing and demonstrating new methods for evaluation. In response