Impact Evaluation of Energy Interventions. David A. Raitzer

       Creative Commons Attribution 3.0 IGO license (CC BY 3.0 IGO)

      © 2019 Asian Development Bank

      6 ADB Avenue, Mandaluyong City, 1550 Metro Manila, Philippines

      Tel +63 2 632 4444; Fax +63 2 636 2444

       www.adb.org

      Some rights reserved. Published in 2019.

      ISBN 978-92-9261-588-8 (print), 978-92-9261-589-5 (electronic)

      Publication Stock No. TCS190113-2

      DOI: http://dx.doi.org/10.22617/TCS190113-2

      The views expressed in this publication are those of the authors and do not necessarily reflect the views and policies of the Asian Development Bank (ADB) or its Board of Governors or the governments they represent.

      ADB does not guarantee the accuracy of the data included in this publication and accepts no responsibility for any consequence of their use. The mention of specific companies or products of manufacturers does not imply that they are endorsed or recommended by ADB in preference to others of a similar nature that are not mentioned.

      By making any designation of or reference to a particular territory or geographic area, or by using the term “country” in this document, ADB does not intend to make any judgments as to the legal or other status of any territory or area.

      This work is available under the Creative Commons Attribution 3.0 IGO license (CC BY 3.0 IGO) https://creativecommons.org/licenses/by/3.0/igo/. By using the content of this publication, you agree to be bound by the terms of this license. For attribution, translations, adaptations, and permissions, please read the provisions and terms of use at https://www.adb.org/terms-use#openaccess.

      This CC license does not apply to non-ADB copyright materials in this publication. If the material is attributed to another source, please contact the copyright owner or publisher of that source for permission to reproduce it. ADB cannot be held liable for any claims that arise as a result of your use of the material.

      Please contact [email protected] if you have questions or comments with respect to content, or if you wish to obtain copyright permission for your intended use that does not fall within these terms, or for permission to use the ADB logo.

      Corrigenda to ADB publications may be found at http://www.adb.org/publications/corrigenda.

      Notes:

      In this publication, “$” refers to United States dollars.

      ADB recognizes “China” as the People’s Republic of China and “Vietnam” as Viet Nam.

      On the cover: Impact evaluation needs to consider the effects of Energy sector interventions on various sectors (photos by ADB).

      Cover design by Joe Mark Ganaban.

       Foreword

      Strategy 2030 of the Asian Development Bank (ADB) emphasizes the creation of knowledge during the implementation of projects and the use of evidence from past operations to design new interventions. The Strategy also envisions a more proactive role for research to backstop ADB’s function as a knowledge provider that helps replicate good project practices across Asia. For these ambitions to be fulfilled, evidence on the intended and unintended effects of interventions needs to be accumulated. Impact evaluation is an important means for generating this evidence.

      To build a broader evidence base on “what works” in development, impact evaluation needs to be mainstreamed across a range of development investments. Presently, impact evaluation coverage has made more progress in the health and education sectors. While sectors dominated by “hard infrastructure”, such as energy, account for far more development investment, the number of impact evaluations on these sectors has remained much more limited. At the same time, energy investments have become much more complex, and have become more oriented toward environmental and social goals over time. Along with these innovations has come an increasing array of behavioral assumptions underpinning interventions, which impact evaluations can help to test.

      Yet, energy sector interventions have special challenges for impact evaluation. Many energy investments, such as powerplants or transmission lines, are “small-n” interventions, which affect large geographies, so that there are insufficient numbers of treated and untreated units for enabling conventional statistical analyses. New powerplants and other infrastructure may also allow improved levels of energy provision, rather than presence versus absence of a service, as the typical context for many traditional impact evaluation techniques.

      This review is intended to help navigate these challenges and offer insights to orient future impact evaluations on the energy sector. It attempts to characterize theories of change associated with many energy interventions, entry points for impact evaluation, impact evaluation challenges, and potential impact evaluation designs. It then reviews the impact evaluation literature to date, in terms of methods applied, interventions covered, and outcomes evaluated, and identifies gaps for future studies to address. In the process, it offers practical examples that might inspire future studies.

      The literature survey finds a rapidly growing body of impact evaluation studies related to energy. Of the 85 studies identified, more than 60% were published after 2012. Impact evaluation designs have also become diversified, with expanding use of more robust methods, such as randomized experiments and regression discontinuity design, as well as greater use of “big data”. Evidence has been offered on a range of effects of electricity access including education, income, health, employment, business performance, and fertility.

      Yet, many evidence gaps remain. Existing impact evaluation effort is mostly concentrated on rural electrification and improved cookstoves, even if most investments are in power generation and transmission. Other important areas of effort, such as energy efficiency, also have little impact evaluation to date. While many studies have evaluated effects of electricity access on education, far fewer have evaluated effects via intermediate services, such as health or agricultural extension facilities, and evidence on firms has remained limited.

      With more big data increasingly available, it is hoped that the creative examples identified in this review can help to inspire new impact evaluation studies that address evidence gaps on energy. This work is more relevant than ever, as energy investments increasingly seek to experiment with innovations, high-level technologies, and behavioral change approaches to ensure sustainable, efficient, and inclusive use of energy. Continued efforts on impact evaluation will be critical to ensure that these experiments can be tested to generate evidence to improve energy operations over time. This review helps to show where future impact evaluation investments can best contribute new evidence and how such studies may be approached.