Economic Evaluation in Education. Henry M. Levin
an overview of the best way to calculate costs—the ingredients method—and three methods of economic evaluation: (1) cost analysis, (2) CE analysis, and (3) BC analysis. Lastly, we explain how we see economic evaluations working in the policy context. As an overview, this material serves as preparation for the greater detail provided in the subsequent chapters.
1.1. Purpose and Goals of the Book
The purpose of this volume is to provide a diverse audience—evaluators, researchers, educational administrators, and graduate students—with a systematic introduction to the use of cost analysis in educational evaluation. Accordingly, the volume has been written with the intention of familiarizing this audience with the ingredients method, a systematic approach to examining costs, and the nature and use of cost-analytic tools, as well as showing how to plan and implement a research study. Economic evaluation refers to a broad set of techniques for evaluation and decisionmaking, including CE, BC, cost-utility (CU), and cost-feasibility (CF). Each type of analysis will be developed separately for consideration, but we will refer to the group of methods taken together as central components of economic evaluation.
The term economic evaluation is deliberately broad. Fundamentally, economics is the study of the allocation of scarce resources, with emphasis on the term scarce. It is not sufficient to investigate how effective an intervention is—or whether one intervention causes a particular outcome. Although these are important and in many cases necessary investigations, they are not fully economic investigations: They do not incorporate the idea of scarcity and in particular do not weigh effects against the opportunity costs incurred to obtain the effects. There are many such examples in education. For example, there is much attention to teacher value added models and how important teacher quality is for learning outcomes. Unquestionably, this is an important area of social science inquiry. But it is isolated from the ultimate question: Is it worth giving up scarce resources to improve teacher quality? What is the opportunity cost of professional development programs to increase teacher quality? By economic evaluation, we mean the class of evaluations that explicitly address the cost of implementing a policy, reform, or intervention or that explicitly compare costs with benefits. In our specific case, these evaluations all relate to education.
The general goal of this volume is to provide the reader with an understanding of how to design and construct an economic evaluation that is valid for educational policy. To meet this overall goal, the intent is to provide the reader with an understanding of how to identify and measure costs using a standard, rigorous approach that is based on the economic principle of opportunity costs, the ingredients method. For CE analysis, the goal is to provide the reader with an understanding of how to specify impacts and effects for the purposes of economic evaluation and combine this evidence on effects with information on costs. For BC analysis, the goal is to assist the reader in how to specify benefits and their monetary values for the purposes of economic evaluation and combine the benefits with costs to produce BC analysis. The final two goals span all types of economic evaluation: One is to emphasize the need for sensitivity analysis; the other is to establish how economic evaluations must link with the requirements of decisionmakers, stakeholders, and policy professionals.
In addition, it is expected that the volume can serve a different need, which will vary among members of our audience. Some educational administrators and evaluators will wish to go beyond learning about the method and its uses to understanding how to actually apply the method to evaluation and administration within their own work settings. For example, an educational administrator may wish to ascertain how to do studies of CE among various alternatives for providing reading instruction or school lunches, or for designing budget cuts. An educational evaluator may wish to learn how to augment a standard evaluation of alternatives with information on costs. While this volume is not designed to prepare such persons to do these tasks in the absence of other training or assistance, it should be considered a necessary step in that direction. By mastering this introduction, an evaluator or administrator should be able to work effectively with a technical specialist on CE or BC analysis or should be able to undertake additional study in mastering the techniques that will be presented.
We have attempted to design the volume so that an individual can utilize it as part of an informal course of self-study or in a formal course on the subject. At various stages, the learner will be introduced to concepts and their applications. These are accompanied by numerous examples drawn from the applied literature, focusing almost entirely on educational examples. Exercises are provided at the end of the chapters to enable the reader to test his or her understanding of the topic that is being covered. Sample answers to selected exercises are provided in Appendix A.
As a point of clarification, we should emphasize one purpose for which this volume—and economic analysis in general—is not intended. This book is not meant to train individuals to “audit the books” of programs or organizations, or gauge compliance with expenditure requirements and the like. The set of concepts and methods that we describe are meant to assist individuals in weighing the costs of interventions against their outcomes and then in choosing the best intervention.
Accordingly, a final word on the purpose of the presentation is important. In these days of do-it-yourself instruction, it is appealing to provide a set of mechanical steps that one can simply follow verbatim in order to learn a new skill. Unfortunately, this approach is not appropriate for training in cost analysis and economic evaluation. Although one can provide a set of principles that can be used for carrying out the analysis, the actual application in any particular setting will require judgments on the part of the administrator or evaluator. Thus, this volume will not be a substitute for a sensitive and judicious effort by the evaluator in applying his or her craft. Rather, it will provide a new set of concepts and analytic tools that can be incorporated into that activity. Although the guidelines for incorporating these new dimensions will be presented and illustrated, the applications will require careful guidance for the analyst or user.
1.2. The Importance of Economic Evaluations
Why should educational evaluators or school personnel be concerned with economic evaluation? The most superficial answer to this question is that reference to such analysis is often an important source of persuasion as a rhetorical claim. By saying that one has compared the CE of different approaches and found a particular one to be most cost-effective, one can often disarm opponents. Unfortunately, this is the principal way in which the terminology is used in the educational sector. This volume will go beyond such banality. CE analysis should be a topic of concern because it can lead to a more efficient use of educational resources—it can reduce the costs of reaching particular objectives, and it can expand what can be accomplished for any particular budget or other resource constraint. Similarly, BC analysis can identify which investments make the most sense and how much society should invest in education. But these results must be derived from application of rigorous methods.
This book emphasizes that a proper assessment of costs and effectiveness or benefits is a necessary element of a serious evaluation. All too often, either costs or effects are considered separately, and any combined inferences may be misleading. For the most part, educational evaluators attempt to establish whether an intervention is effective in attaining some goal, such as raising academic achievement. Business managers and administrators are often concerned with lowering educational costs in order to fit school or district expenditures within a fixed budget. Nevertheless, both parties should share a common goal: They wish to attain maximal school effectiveness for a given budget, or conversely, they wish to attain a given level of effectiveness at a minimal cost. They are unlikely to accomplish this goal if higher effectiveness and lower costs are pursued as independent goals. To illustrate this point, it is useful to consider three examples in sequence.
First, there is the peril of ignoring costs. This peril is nicely illustrated by looking at the development of policies related to smaller classes. During the 1990s, reducing class size was among the most popular and intuitively appealing options for improving schools; it seemed to provide a rare instance of agreement among parents, teachers, administrators, and even politicians. It led to a number of initiatives to reduce class size, especially in the primary grades, with the largest