Planning and Executing Credible Experiments. Robert J. Moffat
book. Bob graciously allowed his notes to supplement my course. Over 18 years, I had developed notes on certain powerful, open‐source software for experiment design and analysis. Bob at Stanford and I, now as professor of energy science at Kyoto University, began exploring how to merge our individual works into a single book. This text is the fruit of our efforts.
This book serves as a generalist guide to experiment planning, execution, and analysis. The text also introduces two powerful, free, open‐source software tools: Gosset to optimize experiment design, and R for statistical computing. This book addresses renewed demands by the public and science community that science be credible. Recently the purported reliability of many landmark medical studies has been undermined due to poor experiment design, execution, or analysis. Separate studies could not reproduce their results. Furthermore, a rash of scientific papers have been retracted due to fraud. Credible science needs a solid new footing.
Audience
We encourage readers to use our text while planning and executing an actual experiment. If the reader already has a problem that must be answered by experiment, that provides the best motivation for each chapter. Each chapter proposes questions that an experimenter will need to ask and answer during each stage of planning and execution.
Major portions of this text have been class‐tested at Stanford in graduate and undergraduate mechanical engineering courses for decades (since the late 1970s). This text has also been class‐tested internationally, serving engineering, physics, chemistry, agriculture, industrial processes, medical, and business students. Drafts of this book have been used for a continuing education course by researchers at the National Renewal Energy Laboratory, among other national laboratories and industrial laboratories.
Our book (hereafter referred to as M&H) is designed to be a time‐proven, single‐source guide for an experimentalist, from initial conception of a need, through execution of the experiment, to final report. Our book will stand alone in the lab, yet introduces researchers into specialist texts.
Accompanying Material
The open‐source software referenced in this text is free on the internet. The software, along with example data from the text, is additionally provided on an online website for this text.
Recommended Companion Texts
Our text forms a close companion with Hugh W. Coleman and W. Glenn Steele, Experimentation, Validation, and Uncertainty Analysis for Engineers, 4th ed., published by Wiley (hereafter referred to as C&S) (2018). Coleman was a student of Moffat's. C&S section 1.2, “Experimental Approach,” outlines our book in one page.
Statistics for Experimenters, 2nd ed., by George E. P. Box, J. Stuart Hunter, and William G. Hunter (hereafter referred to as BH&H), is a Wiley classic (2005).
Response Surface Methodology, 4th ed., by Raymond H. Myers, Douglas C. Montgomery, and Christine M. Anderson‐Cook (hereafter referred to as MM&AC), is also published by Wiley (2016).
How Is This Book Used for Teaching?
Our book has been used in Experiment Design courses as well as in diverse laboratory courses. Students in a variety of fields, including physics, engineering, chemistry, genetics, economics, medicine, and environmental studies, have used this book. Our text has also found a home directly in the lab (independent of classroom instruction) as a guidebook. It is written for self‐study and continuing education; as such, it has been the text for short courses at national labs.
Acknowledgments
We thank N.J.A. Sloane and R.H. Hardin for developing the Gosset Program for Design of Experiments. We deeply appreciate their releasing Gosset to the public domain in 2018. We expect that researchers will benefit from the capabilities of Gosset for decades to come.
We are grateful to the thousands upon thousands of users of the R statistical language, contributors and researchers worldwide who have evolved R into a powerful tool for statistical analysis, modeling, and plotting. Thank you all for promoting this open‐source, free software.
Robert thanks and dedicates this book to Lou London of Stanford University who taught me, inspired me, mentored me, and changed my life, opening doors across the world.
Roy especially thanks Helen L. Reed, William S. Saric, and William C. Reynolds for launching me into this most fascinating field, experimental thermal‐fluid physics. R.J. Hansen and R. Rollins at the U.S. Naval Research Lab introduced me to advanced instruments and signal processing. Stanford colleagues and students ensured that all angles were considered and defensible.
Colleagues at Kyoto University (京都大學), IHI (石川島播磨重工業株式会社), Science University of Tokyo, NAL, and Tokyo University provided a convivial environment while I worked in the aerospace engine industry and energy science. Together we braved the national impact of the Fukushima tsunami. Colleagues and students at Handong (韓東大學校, ) and LeTourneau provided valued support and feedback.
D.T. Kaplan and his books are outstanding for training statistical modeling.
We thank science editor A. Hunt of Wiley who paved the way for us as authors. S. Benjamin assembled documents. S. Brown ensured protocol. Ashwani Veejai Raj and her team converted diverse formats into typeset proofs. Eagle eyes of Cherrine R., Elaine H. and Lucy K. spotted stray prey. L. Poplawski corralled our project into final form. We thank you all.
Bob Moffat and his wife, Karina, have been constant encouragement as this work progressed. Whether on the other side of the world or in their dining area, they have been most generous.
My family and their enduring support are treasure beyond measure.
About the Companion Website
This book is accompanied by a companion website:
www.wiley.com/go/moffat/Planning
Scan this QR code to visit the companion website.
The website includes:
Problems
Sample data sets
Archived sources to install Gosset and R
Solution Manuals
1 Choosing Credibility
No one believes a theory except its originator.
Everyone believes an experiment except the experimenter. 1
The decision to design a credible experiment sets you on a path to research with impact. Along this path you will make many decisions. This book prepares you to anticipate the choices you will face to plan and achieve an experiment that you the experimentalist also can believe.
There are two kinds of material to consider with respect to experimental methods: the mechanics of measurement and the strategy of experimentation. This book emphasizes the strategy and tactics of experiment planning.
The fact is