Planning and Executing Credible Experiments. Robert J. Moffat
978‐0618551057.
20 Thomson, W. (1883). Wikiquote. https://en.wikiquote.org/wiki/William_Thomson
21 Tinsley, J., Molodtsov, M., Prevedel, R. et al. (2016). Direct detection of a single photon by humans. Nature Communications 7: 12172.
22 Woit, P. (2006). Not Even Wrong: The Failure of String Theory and the Continuing Challenge to Unify the Laws of Physics. Basic Books. ISBN‐13: 978‐0465092765.
Homework
1 2.1 Prepare a lab computer for dual boot: Windows and the Linux operating system. Or …
2 2.2 Prepare a dedicated lab computer for the Linux operating system. Or Exercise 2.1.
3 2.3 Following the guide in Appendix D3, download and install Gosset, public domain, open source, and free. Please consider this software tool essential.
Notes
1 1 This quotation is from a Feynman lecture at Cornell in 1964. A select portion of the lecture can be viewed at https://youtu.be/OL6‐x0modwY. A lengthier quote is given in Pomeroy (2012).
2 2 Rather than divide a complex number into real and imaginary parts, shall we describe it as revealed and concealed parts? Our reasoning is that both parts of a complex number have reality in our universe. For example, in dynamic systems, the revealed part corresponds to position and the concealed part corresponds to velocity. Is velocity just as real as location? Of course. We, you and us, can see the location of a car relative to its surroundings. The speed (scalar) is revealed to us via the speedometer. The change of velocity is revealed to us via our inner ear. For deeper insight, Roger Penrose has provided a most compelling motivation for complex numbers in his book Road to Reality (2005, chapters 1 through 14).
3 3 The superiority of the cooking profile trajectory is such that I (RH) witnessed a 50-person block of Asian tourists who each traveled to Japan primarily to buy a rice cooker to bring home to their country. There was insufficient room in the overhead compartments for all the rice cookers, so the flight departure was delayed 40 minutes as all the cookers were relocated to the checked-baggage compartment.
4 4 Dr. Loyd Withrow, GM Research Labs, ca. 1953. Personal communication.
5 5 Thank you to Dr. Ioannidis for permission to include the lists from his articles.
6 6 Penrose trained Stephen Hawking; together they wrote a number of landmark articles on the nature of the universe. Penrose's book Road to Reality (2005) highlights the math that underlies various areas of physics. His presentation of the reality and necessity of complex numbers is the best I've (RH) read.
3 An Overview of Experiment Planning
My background is mainly in research and development experiments in heat transfer and fluid physics. When I think of planning an experiment, I think about wind tunnels, heat exchangers, temperature, and flow control. I have tried to generalize my experience, but my background certainly colors my outlook.
Experimental work is expensive. Although costs vary for different fields and situations, the time costs are all similar: all laboratory work runs in real time, an hour for an hour, and there are no short cuts. It is important that experiments be well conceived, well executed, and well documented.
The purpose of experiments is to produce provably accurate data that answer an agreed‐upon question about the behavior of a system. This is the kind of experiment I wish to discuss. Some of the important ideas will be transferable to other types of experiment, but the main thrust here is to deal with R&D experiments with tangible, numerical objectives.
The planning of such an experiment is necessarily iterative. The process starts with a tentative plan: a first impression of the goal, a plausible experimental approach, a possible suite of instruments, a tentative set of tests to run. This plan must then be challenged: will it produce the desired information with acceptable accuracy? Then the plan is refined, sometimes by improving the statement of objectives, sometimes by selecting a better approach, sometimes by improving the instrumentation.
3.1 Steps in an Experimental Plan
The general steps in an experimental program can be summarized in the following outline:
1 Assignment.
The Iterative Loop
1 Determine the objectives.
2 Select the experimental approach.
3 Parametrically design an apparatus.
4 Design apparatus hardware.
5 Construct and install apparatus.
6 Design analysis software; debug with fabricated sample data.
7 Perform shakedown, debugging, and qualification runs.
The Execution
1 Collect data.
2 Reduce data and analyze.
3 Report.
Seldom is a successful research experiment designed on a once‐through basis. This is not surprising when you think about the amount of scratch paper usually generated in trying to develop an original analysis. Alternative experimental approaches must be investigated and trade‐offs made between accuracy and convenience, range and speed, etc.
3.2 Iteration and Refinement
Research echoes sailing into uncharted territory. Sailboats cannot sail directly into the wind. The skipper must tack into the wind, repeatedly aiming right then left, iteratively correcting course.
This book takes an iterative approach to experiment planning. For example, it may not be clear how to execute some of the steps until later, when their background material has been developed. Some of the steps themselves may not even make sense until the background material has been developed. And some of the background material we ask for won't make sense until the need for it has been established. This circularity is typical of large‐scale projects with interactions: They cannot be studied sequentially, they have to be approached integrating “all at once” and iterating.
We emphasize the iterative nature of experiments. What seems plausible at first may not prove acceptable later. Sometimes a preliminary “exploratory” experiment is a good investment – to test an approach. It may show that the original concept of the experiment is not a good one. You likely will iterate the experiment itself, as in Figure 1.1. Details of the planning steps and their objectives are dealt with in subsequent chapters.
An additional issue of risk assessment warrants early treatment because it affects every decision in the planning process.
3.3 Risk Assessment/Risk Abatement
Consider the following risks:
The data may not answer the motivating question.
The