Economic Evaluation in Education. Henry M. Levin
school revenues. Narrowing the question to which schools are to be closed is to rule out options that may be more appropriate when economic criteria are used. That is, there can obviously be no economic analysis among alternatives that are not considered.
To select the most efficient program or intervention, all relevant alternatives should be identified and considered. The alternatives for addressing particular problems are those potential interventions that might respond to the problem and improve the situation being addressed. It is important to ask whether all the pertinent alternatives have been placed on the agenda for consideration. Obviously, the classes of alternatives that ought to be considered are those that are most responsive to the problem. Again, this will require a sensitive search for ways of meeting the challenge that has been posed. Although one may wish to draw upon traditional responses as well as those that other entities have used in facing similar problematic situations, one should not be limited to these. In fact, sometimes they may not be the most responsive approaches.
In conducting an economic evaluation of alternative interventions, one must ask not only whether the alternatives that will be considered are responsive to the problem but also whether the “responsive” ones have been included. Often, both administrators and evaluators will rule out alternatives before analyzing them when such options are politically sensitive. That is, the pragmatic aspects of daily life suggest that one avoid pitched political battles by keeping politically sensitive issues off the agenda if at all possible. While one can appreciate the pressures on both evaluators and administrators in this regard, there are two reasons that all of the relevant alternatives should be analyzed.
First, it is a matter of professional integrity to provide information on all of the pertinent alternatives, while letting the decisionmaking and political processes eventually determine the choice among them (Ross, Barkaoui, & Scott, 2007). If those processes are not adequately informed about possible responses, they can never consider the costs and impacts of many of the pertinent alternatives. There is an appropriate place for analysis and one for decisionmaking. If certain alternatives are precluded from consideration by their political sensitivity, then the political and decisionmaking processes have taken place before the information and analyses have been derived. Clearly, the two stages are interrelated, but good decisionmaking should be based upon informed choices rather than ones that eliminate potential options before they are ever analyzed and considered.
A second reason for considering even those alternatives that are politically sensitive is that such sensitivity or opposition may be dependent upon circumstances. That is, while some alternatives are indeed “untouchable” in the normal course of events, they may become salient for consideration under more dire circumstances. If a school district is facing serious budgetary problems, it must consider all possibilities that would reduce the budget. If student proficiencies in certain academic areas are woefully inadequate, then a wide range of programs for improvement begin to enter the realm of consideration. It is important to consider the strengths and weaknesses or the costs and effects of selecting from all of the pertinent alternatives. It should also be borne in mind that the retention of existing practices is always an alternative.
Indeed, this leads to a final comment on alternatives. Cost analysis is premised on the view that decisionmakers have choices. The objective is to make the best selection from competing alternatives. Economic evaluation is done in order to aid in the selection among alternatives. If there are no alternatives, there is no point in doing an analysis. That is, no matter how competent the evaluation, it will simply lack usefulness if one cannot do anything with what is learned. Equally importantly, the analyst should evaluate a given intervention against a reasonable alternative that might otherwise be implemented. The evaluation should not compare the intervention against “nothing” if there is a reasonable expectation that students would receive “something”; the intervention should be compared against the other “something.” (The analogy here is with drug trials where a new drug is compared against a placebo rather than a generic version of the drug.) In fact, in order to properly perform an incremental analysis, an economic evaluation should focus as much inquiry on the comparison group as it does on the treatment group.
Ultimately, the goal of an economic evaluation is to help policymakers decide on an alternative. It is to help them choose among alternatives. Of necessity, it can only help them choose among alternatives that have been the subject of the analysis and the type of analysis chosen. Cost-effectiveness (CE) analysis of six school-based reading programs can only provide information as to which reading program is the most efficient; it cannot determine whether school-based programs are more efficient than parental engagement reading programs. Also, it cannot determine whether a district would get higher benefits from investing in math programs instead of reading programs. All research is bounded by the questions included in the analysis, but not all research is so directly intended to influence decisions. The goal of the economic evaluation of reading programs is to help decisionmakers choose one.
2.2. Taking Account of the Audience and Perspective
When identifying the problem and the alternatives that might address the problem, it is important to be clear about the audience or audiences for whom the analysis will be done. The audiences include one or more groups of stakeholders—that is, individuals or institutions with an interest in the outcomes of the evaluation process. It is helpful to think of a primary audience and a secondary audience. The primary audience is generally the decisionmaker (and the clientele whom he or she represents) who has requested the analysis. The secondary audience consists of those persons and groups who will also draw upon the analysis.
Since the analysis is being prepared explicitly for the primary audience, it is important that it meet the specifications of that audience. For example, if one is requested to do a cost-feasibility (CF) study of using educational television in the school curriculum, it is important to ascertain exactly what is behind the request. Is there a specific technological or curriculum approach that the decisionmaker has in mind, or is the charge to be concerned with the costs of a wide variety of approaches, from selected courses to full curriculum coverage? These details can be worked out through dialogue and specification of the issues, and the analysis and report should be written with the needs of the primary audience in mind. Nevertheless, there may be tension if the decisionmaker places certain alternatives beyond consideration even though the alternatives may be in the best interests of the constituents represented by that decisionmaker. For economic analysis, there is an extra tension: With an impact evaluation, there is a risk that the intervention will not be effective; with an economic evaluation, there is an added risk that, even if it is effective, the intervention will not be found to be efficient (Levin, 2001). Hence, it is important to structure the analysis so that all reasonable alternatives are evaluated in the same way and so that the research conclusions are not predetermined.
Often, however, economic reports are read by secondary audiences who wish to use the information to inform decisions regarding replication or scale-up in another setting. If a study will be restricted to its primary audience, one need not be concerned about secondary audiences. However, if a secondary audience is likely to utilize the study, it is important to be clear about what is possibly generalizable to other settings and what is not. For example, in reviewing the costs of instructional programs, a school district may generate cost analyses that exclude resources contributed by the state or other levels of government or by volunteers. For purposes of decisionmaking in the district under scrutiny, this omission may be understandable. But if other school districts attempt to apply those cost analyses, they may discover that they have allocated far too few resources to support such instructional programs. Not every other school district in that secondary audience will have equal access to governmental or volunteer resources. In general, it is preferable to report all costs such that any audience can identify the resources appropriate to their situation no matter who is providing them.
Potentially, there are many audiences. For example, a district may be interested in improvements in teaching English and reading to students from non-English-speaking backgrounds. Among the alternative policies are English as a second language (ESL), bilingual instruction, and total immersion in English. Before performing CE analysis to determine which policy to recommend, the analyst might identify several primary and secondary audiences. The primary audiences—that