Impact Evaluation of Energy Interventions. David A. Raitzer
Research and Regional Cooperation Department
Asian Development Bank
Acknowledgments
This publication reflects the contributions of many individuals within and outside of the Asian Development Bank (ADB). It has been produced under the overall guidance of Edimon Ginting, Deputy Director General of the Economic Research and Regional Cooperation Department and Rana Hasan, Director of the Economic Analysis and Operational Support Division. The publication has also benefited from overall orientation by ADB’s interdepartmental Impact Evaluation Committee chaired by Bernard Woods, Director of the Results Management and Aid Effectiveness Division. David Raitzer, Economist, Economic Analysis and Operational Support Division; Nina Blöndal, independent consultant; and Jasmin Sibal, Economics Officer, Economic Analysis and Operational Support Division, authored the contents.
Sakiko Tanaka provided valuable insights and contributions to early versions of the report. Independent consultant Howard White and Christopher Edmonds also provided valuable suggestions. Additional inputs and contributions were provided by ADB consultants Daryll Naval, Denise Encarnacion, and Reneli Gloria. Administrative support has been provided by Lilibeth Poot, Jindra Samson, Amanda Mamon, Gee Ann Burac, Rica Calaluan, Roslyn Perez, and Glennie Amoranto.
Valuable peer reviews have been provided by Sujata Gupta, Ryotaro Hayashi, Francesco Tornieri, and Aziz Yusupov. Their insights have helped to improve the relevance of the report.
Copy editing was performed by Tuesday Soriano and layout was performed by Joe Mark Ganaban. This book is produced under Regional Technical Assistance 0012 for Developing Impact Evaluation Methodologies, Approaches and Capacities in Selected Developing Member Countries.
Key Messages
Traditional economic assessment of energy interventions depends on many untested assumptions. Economic appraisal of energy interventions typically relies on the use of partial and general equilibrium models. In those models, behavior is assumed to follow mechanistic assumptions regarding demand elasticities and substitution patterns. For example, traditional energy sources are simply assumed to be replaced by modernized sources when available, and efficiency improvement is assumed to lead to lower energy consumption. However, these all depend on untested assumptions about how energy is used and how energy users behave. Understanding these behavioral aspects is critical to making energy investments more effective.
Impact evaluation can help to test key energy sector assumptions and effects. The approach can test basic assumptions about substitution effects of energy projects, as well as the effects of a range of energy policies. It can reveal which aspects of energy efficiency programs are effective. A range of outcomes for households, ranging from time use, to leisure, social capital, schooling, disease burden, and employment can all be tested, as can effects on firms and the environment.
Impact evaluation is increasingly relevant to energy programs. Energy investment is becoming more oriented toward environmental and social objectives. The sector has increasing investment that seeks to shift energy consumption behavior to be more efficient, fuel choices to be cleaner, energy usage to be more productive, and energy distribution to be more equitable. All of these effects depend on behavioral responses to energy interventions, which impact evaluation can help to test.
Impact evaluation of energy interventions is rapidly growing from low levels. Energy, especially in the power sector, has been one of the main sectors receiving official development assistance. Historically, very little impact evaluation has focused on energy. However, this is changing quickly. This study identifies 85 impact evaluations of energy projects, policies, and other investments in developing countries, of which more than 60% were published after 2012. Even so, this is a fraction of the number of studies available for the social sectors. Energy projects are increasingly complex and dependent upon behavioral assumptions, so that only a small fraction of these assumptions have been tested through impact evaluation for a limited range of interventions, in some contexts.
Impact evaluations have offered important insights on a range of effects of energy interventions. These include significant effects of electricity access on energy use, income, consumption, education, gender disparities, health, and even fertility. However, much of the evidence is from a very limited set of interventions and contexts, and there is much variation among findings. The reasons for these variations are yet to be fully understood. Most impact evaluations have focused on the effects of rural electrification on households. The smaller set of impact evaluations conducted on electricity reforms generally find fewer significant effects.
Impact evaluation of energy interventions often requires a creative approach. There are special challenges for impact evaluation of energy projects, especially large generation and transmission infrastructure that affects large areas of countries. At the same time, several methods show promise for energy sector applications but have not been used frequently. These include synthetic controls, regression discontinuity designs, and encouragement designs.
Randomized evaluations have promise to reveal new insights on energy interventions. Although large energy infrastructure often cannot be randomly assigned, encouragement designs can introduce random variation in the use of energy, and incentive programs for demand-side energy efficiency and use of clean energy can often be randomized. These designs have untapped potential to enable testing of an expanded array of assumptions and interventions.
“Big data” can enable new impact evaluation possibilities on energy. High frequency, geospatial, and “crowdsourced” data are increasingly available, and many of these data sets are relevant to energy. For example, remote sensing can now detect luminosity and pollution, and billing data can be harvested to understand grid electricity use. These developments expand the numbers of variables and outcomes that can be included in analysis.
There are important evidence gaps on energy, and there is ample scope for further research. Only a limited range of interventions has been assessed. Investments in power generation and transmission for improved electricity capacity and reliability, incentives for clean energy deployment, smart grids, renewable energy deployment, energy efficiency measures, fuel substitution, and energy interventions outside of electricity and improved cookstoves have received little attention. Evidence is also lacking for many intermediate channels, such as effects on public services, as well as longer-term effects on health and employment.
The development community can help address evidence gaps. As leading supporters of energy investments in Asia, development banks, such as the Asian Development Bank, are well-situated to contribute to producing new knowledge on what works in the sector.
1. Importance of Impact Evaluation in the Energy Sector
Energy is a rapidly evolving area of development assistance. Whereas energy access through conventional infrastructure has been a traditional emphasis, development assistance increasingly recognizes the potential of innovative technologies and policy reforms to contribute to a wider range of outcomes related to sustainability and social development. Yet, even as the objectives of the field have expanded, the existing body of evidence of what works in the energy sector has remained limited.
Many economic analyses have been used to assess the economic gains from energy projects through modeling that relies heavily on assumptions (ADB 2013a, 2017). Until recently, more rigorous studies examining the effects of energy interventions were mainly macroeconomic and based on time series or modeling using partial equilibrium optimization or computable general equilibrium (CGE) techniques (Torero 2014). Most of these models are ex ante, and attempt to compare alternative energy sector policies, in terms of implications for economic growth and/or environmental emissions.
Although useful for forecasting and simulating options, such studies assume, rather than identify, causal relationships and behavioral responses to interventions. Impact evaluation, on the other hand, empirically measures the causal effects of interventions on outcomes of interest, while minimizing