Economic Evaluation in Education. Henry M. Levin
open online courses (MOOCs) may necessitate investment in an expensive technology infrastructure before any students have enrolled (Bowen, Chingos, Lack, & Nygren, 2014). For some interventions, there may be reorganizational costs that are not obvious. For example, implementation of Success for All requires cross-grade grouping—that is, putting students from different grades in the same classroom; this grouping may have implications for how the school day is organized (Quint, Zhu, Balu, Rappaport, & DeLaurentis, 2015). Finally, some programs may have substantial induced costs such that the biggest cost item is not the intervention itself but the extra resources that flow from implementing the program. A series of examples, showing how costs change as students progress through community college at different rates, is modeled by Belfield, Crosta, and Jenkins (2014).
Relatedly, cost analysis can be applied to check for fidelity of implementation of an intervention. For example, high-quality preschool programs are expected to have a trained lead teacher; the cost analyst can verify if sufficient funds were allocated to ensure that this is affordable. Subject to the caveats just noted, the analyst can compare actual resource use with an ex ante budget to see whether the program resources match the intended design.
The second intrinsic value of cost analysis is that it helps the researcher with prediction and modeling. It is important to recognize that costs are estimated, just as effects are in an impact evaluation. The researcher is estimating an expectation based on the sample units of the intervention. In some cases, the analyst will want to estimate the actual costs of the program as it was delivered in a particular setting for a specific group of students. It is more likely that the analyst will prefer to estimate expected costs—that is, the best prediction of what the program will cost if implemented again. These two estimates will be different: For expected costs, the analyst will use actual ingredients but general prices; for actual costs, the analyst will apply the actual (or as close as possible) prices each site paid for a particular ingredient. This latter approach is similar to a budget approach and so must be applied cautiously.
The use of expected costs allows the analyst to more accurately model costs. One concern for education research is how to take a successful small or pilot intervention and implement it at scale with the same level of success (Quint, Bloom, Black, Stephens, & Akey, 2005). For instance, the policymaker might mandate a class size reduction policy such that all classes must be 10% smaller. What resources would be required to implement such a mandate? It may require proportionately (10% extra) teachers and classroom space but only a smaller proportion (less than 10% extra) of administrative faculty and school facilities. With information on the costs of reducing class size by various proportions, it is possible to predict average cost as the intervention expands (i.e., economies of scale). For example, in their analysis of a schoolwide positive behavioral support intervention, Blonigen and colleagues (2008) estimated significantly lower costs for districts with 10 schools relative to districts with just one school. Deming, Goldin, Katz, and Yuchtman (2016) showed the use of online learning in higher education can shift the average cost function downward—that is, lower the cost per student.
Cost information can also be used to model different versions of the intervention. With data on how programs vary in their resource use, the analyst can see if there are alternative mixes that yield a similar cost per participant. For example, a preschool program might be able to hire four different levels of instructor (e.g., master teacher, lead teacher, adjunct, or teacher aide). Cost analysis can help determine how these instructor types can be combined in a way that would yield a given cost per preschooler. For a thorough example using cost functions for child care, see Blau and Mocan (2006).
Finally, information on the distribution of financing of resources across stakeholder groups is useful for policy modeling. Typically, costs are distributed among various social stakeholders: These might include the federal government, school districts, schools, parents, local businesses, philanthropic agencies, and so on. By understanding the distribution of the financing of costs, we are in a better position to analyze how stakeholders might support or oppose an educational program. For example, the economic value of college is typically found to be very high, prompting questions as to why students do not invest more (Avery & Turner, 2012). But if this value is shared across the student and society, then the student may not have an incentive to stay in college for longer (Trostel, 2010). Alternatively, an intervention may make economic sense to a school district because it does not have to fully fund the intervention: With matched federal funding, a program need only be modestly effective to be worthwhile to the school district.
3.6. Conclusions
This chapter examines costs conceptually. We do so to make several key points for cost analysis.
The first point is that resources should be valued in terms of their opportunity cost. This simple idea is very powerful in two ways. First, it forces us to always think about the scarcity of resources: No teacher’s time is free, nor is any student’s time free. Second, it forces us to think about valuation in two steps: One is identifying each ingredient; the next is putting a value on that ingredient that reflects its opportunity cost. Thus, opportunity cost is the foundation of the ingredients method discussed in the next three chapters.
The second point is that the costs of an educational intervention are “all the resources needed to make the intervention work.” We emphasize all resources because educational programs typically draw on contributions from various sources: different levels of government; different tiers within a school system; a range of community resources; and, of course, changes in how students spend their time. A systematic and comprehensive way of thinking about costs—related to the theory of change—is needed to accurately value each program.
Third, we want to dispel the notion that cost analysis is a simple exercise using budgetary information. A cost analysis is an actual research endeavor in the same way that an impact evaluation is. As we illustrate in upcoming chapters, this necessitates a research design for data collection and analysis that is methodologically valid.
Finally, we hope we have encouraged the reader to devote more attention to cost analysis within the field of education research (see Chandra et al., 2011). A rigorous cost analysis generates a lot of information about each intervention, and this information can be useful in many different ways for educational decisionmakers.
In the next set of chapters (Chapters 4–6), we apply these concepts to cost analysis using the ingredients method.
Discussion Questions
1 What is meant by the term cost when used in cost analysis? What is meant by “incremental” cost? Provide a hypothetical example.
2 When might it be inappropriate to use a market price for employing a particular ingredient in future replications of an educational evaluation?
3 In what ways can understanding a program’s logic model or theory of change help you design a cost study of the program?
4 What characteristics of budgets make them inappropriate sources for estimating costs?
Exercises
1 What are the “costs” associated with the following situations?The school your children attends is temporarily closed because of building violations.The district decides to reduce the number of schools a family can apply to for high school.The school sponsors an outdoor party for the student body that destroys a major portion of the lawns and shrubbery.A rise in school crime and vandalism requires that some teachers be used to patrol the campus rather than teach.The birth of a child places heavy demands on your family schedule so that you must defer the completion of courses for a master’s degree.
2 You observe a mentoring program over several years and across several sites. The program is 1 year long, and you observe very little year-over-year and site-by-site variation in implementation and costs. You are therefore able to estimate that programs that serve few to no students (e.g., during start-up years) cost about $10,000, sites that serve 25 students in a year cost