Business Experiments with R. B. D. McCullough

Preface

      Rigorous experiments are rapidly becoming a critical tool for businesses. While experiments have been a popular tool in engineering for decades, driven largely by the quality movement of the 1980s, experiments are rapidly coming out of the engineering lab to be deployed more broadly across business operations. Companies today are using experiments to solve a broad range of problems from determining whether improved training results in better employee productivity and retention to gauging whether store renovations result in higher profitability, to determining which subject lines in a promotional email attract customer attention, to assessing which behavioral nudges are most likely to get utility users to shift energy consumption to nonpeak times, and to deciding which color to use for the checkout buttons on a website. Articles in the business press such as those by Davenport (2009), Anderson and Simester (2011), and Thomke and Manzi (2014) have made the case that experiments are a key component of “data‐driven” business and companies are taking note and incorporate experiments into their day‐to‐day operations. With this rapid adoption of experimentation, there is a need for a comprehensive resource for business leaders and analysts who want to use experiments to understand how business systems work and solve tactical business problems.

      Yet, material on business experiments remains siloed within specific business functions. For example, there are dozens of books and articles on website testing that focus intently on experimentation with websites and never mention that the same basic ideas – randomization, measurement, and comparison – can be applied in other business domains. Our goal for this book is to step back from this highly tactical approach and give business students a broad, theoretically grounded understanding of the role of experiments in business, providing them with a practical foundation for designing and analyzing experiments that will be useful to them throughout their careers.

Of course, experiments have long been a foundational tool in engineering, physical sciences, psychology, medicine, and statistics, and excellent books on how to design and analyze experiments are available within each of these fields, yet none of these books provides the right coverage for business students. The literature on experiments in the physical sciences and engineering provides great depth on multivariable test design, yet often focuses on situations where the experimenter is studying a physical process in a lab setting, where everything can be measured and controlled and error can be reduced nearly to zero – conditions that seldom hold in business. The classic textbook on experimental design for engineers by Montgomery (2017) naturally assumes a level mathematical skill that is not common among business students. Psychologists like Campbell and Stanley (1963) focus more on experiments with human subjects, raising many of the same issues that come up in marketing and human resources experiments, yet they focus intently on using experiments to develop theory and so cover issues like construct validity that are less relevant in the types of tactical experiments that business uses to drive day‐to‐day decisions. Useful discussions of the practicalities of running experiments in the field have appeared