Economic Evaluation in Education. Henry M. Levin
resources being applied.
Moreover, estimates of costs alone are important for another reason: They are a guide to affordability. CF analysis refers to the method of estimating only the costs of an alternative in order to ascertain whether or not it can even be afforded. If the cost of any alternative exceeds resources that are available, there is no point in doing any further analysis. As a concrete illustration, one might view the situation of compensatory education, in which a specified amount is available for augmenting the education of each disadvantaged child. If this amount is $400 per child, then any alternative that violates this constraint would not be feasible. Nevertheless, even this limited scope is helpful for policymaking. CF represents a limited form of analysis that determines whether or not alternatives are within the boundaries of consideration. It cannot be used to determine which affordable options should actually be selected. As another example, we can imagine a community that believes it is equitable to spend the same amount on each grade; early interventions in first grade, even if they are found to be highly effective, might be rejected on equity grounds because they involve giving disproportionate resources to a single grade. See Example 1.1 for CF analysis of class size reduction.
Example 1.1 Cost-Feasibility Analysis of National Class Size Reduction
Class size reduction has been a popular policy for improving educational outcomes, and the evidence that it increases achievement in the early elementary school grades is compelling (Krueger, 1999). A school district—faced with accountability pressures to improve performance—might consider reducing class size. But even before considering whether class size reduction will boost test scores, a much simpler and more basic question is whether such a sweeping plan could be feasibly implemented in light of current and future budget constraints. Would the costs associated with a national class size reduction be prohibitive?
Dominic Brewer and his colleagues (1999) estimated the expected costs of different approaches to national class size reduction in Grades 1 through 3. To yield a baseline cost, this estimation required a number of assumptions about the scope and design of the policy. Their baseline assumptions were as follows. First, there are three possible targets for class size reduction: 20, 18, and 15. Second, class size reduction is implemented districtwide (such that class size reduction is an average per district, with flexibility per classroom and per school). Third, the policy applies uniformly to all students such that there is no targeting of the policy toward specific groups such as high-poverty schools or districts. Finally, and most importantly, only operational costs are considered, such as salaries and benefits of teachers, aides, and administrators, as well as instructional materials and supplies. No consideration is given to the costs of facilities and infrastructure and no cost is given to the process of transitioning from larger to smaller classes. If these costs were included, the overall cost of class size reduction—the actual cost—would be considerably higher. Upon arriving at their baseline estimates, they varied several key assumptions in order to observe the sensitivity of cost estimates. Their baseline estimates are given in the following table.
Source: Brewer et al. (1999, Table 2).
Note: Adjusted to 2015 dollars.
In 2015 dollars, the cost of operating the education system with class sizes of 20 in Grades 1 through 3 would be an additional $3.19 billion each year (or $280 extra per pupil). Lowering class size to 15 would result in substantially higher total and per-pupil operational costs of $16.57 billion and $1,470, respectively. Leaving aside transition costs and the resources required for facilities, these cost amounts raise questions as to whether reducing class size meaningfully is affordable.
With sensitivity analysis, the authors can tell us what makes the biggest difference to costs. For example, class size reduction is less costly if the targeted levels of class size must only be met on average across the entire state, rather than across districts. On the other hand, it is more costly if average class sizes in each school must meet targeted levels. Moreover, the policy is much less costly if it is targeted only at schools with large numbers of high-poverty students.
Cost-feasibility (CF) estimates cannot inform decisionmakers as to whether class size reduction is a socially desirable investment in absolute terms or whether it is relatively more desirable than another investment. This can only be accomplished by weighing the costs of class size reduction against its benefits or effectiveness. Nevertheless, the cost estimates can tell decisionmakers whether national class size reduction is feasible within the current set of budget constraints. This information is especially useful in situations where the resource requirements are not easily observable (e.g., when the school might not know how many new teachers to hire or how much it will cost to reorganize the school schedule).
Source: Adapted from Brewer et al. (1999).
1.3.2. Cost-Effectiveness Analysis
CE analysis is an evaluative technique that compares policies or programs based on their ratios of costs to educational results on a quantifiable (but not monetized) effectiveness measure (Boardman, Greenberg, Vining, & Weimer, 2011, Chapter 18; Levin, 1975). This analysis allows us to take account of both the costs and effects of selecting alternatives, making it possible to choose those alternatives that provide the best results for any given resource outlay or that minimize the resource utilization for any given outcome. For example, we can compare alternative interventions for improving academic performance in mathematics or reading to consider the interventions that show the largest achievement gain relative to their costs.
Standard approaches to evaluation are more limited because they primarily focus on the effectiveness of alternatives, such as impacts on test scores, socioemotional skills, high school graduation, and so on. Presumably one need just choose those interventions that show large effect sizes over those that show small effect sizes. However, the absence of cost information means that such results should never be used for decisionmaking in isolation. For example, if one of the interventions is associated with an effect size on achievement of 0.6 and another of 0.4, it does not necessarily follow that the first of the interventions is the superior one. What if the cost of the first intervention is $400 per student and the cost of the second is $200 per student? For any given budget, the overall effect of spending it entirely on the second intervention may improve achievement far more than spending it on the first alternative. By combining cost information with appropriate measures of effectiveness, we are able to use resources more productively and improve educational outcomes with given resources. Policy decisions in the public sector must be based increasingly upon a demonstrated consideration of both costs and effects.
Effectiveness can be measured in various ways, depending on the needs of the decisionmaker. For example, alternative interventions can be evaluated on the basis of their cost for raising student test scores by a given amount, or the cost for each potential dropout averted, or the cost per instance of conduct disorder. Moreover, the education system can serve to effect broader societal change: the installation of water fountains in schools can reduce obesity, for example, by reducing the consumption of sugary beverages (Muckelbauer et al., 2009; Schwartz, Leardo, Aneja, & Elbel, 2016). From a decision-oriented perspective, the most preferable alternatives would be those that show the lowest cost for any given increase in the selected effect. By choosing the most cost-effective alternative, we free up resources that can be invested in other aspects of education (or in another endeavor). However, CE requires that (a) only programs with similar or identical goals can be compared and (b) a common measure of effectiveness can be used to assess them. These effectiveness data can be combined with costs in order to provide a CE evaluation that will enable the selection of those approaches that provide the maximum effectiveness per level of cost or that require the least cost per level of effectiveness. For an illustration of this, see Example 1.2.
Example 1.2 Cost-Effectiveness Analysis of Primary School Investments in Northeast Brazil
The states that form northeast Brazil are among the poorest areas in the world. In the 1980s, many children did not even attend primary school. Of those children who did, schools were often attempting to provide education without many basic resources, including infrastructure, classroom