Planning and Executing Credible Experiments. Robert J. Moffat
compared with String Theory. Theoretical physicist L. Smolin raised an alert in his book The Trouble with Physics: The Rise of String Theory, the Fall of a Science and What Comes Next (2006). Noting that physics was rich in alternative “promising new directions,” Smolin wrote to promote other areas of theoretical and experimental physics. Others noticed as well. P. Woit wrote his critique Not Even Wrong: The Failure of String Theory and the Continuing Challenge to Unify the Laws of Physics (2006).
Roger Penrose6 took on three popular areas of physics, expressing similar concerns. Penrose's recent book (2016) is Fashion, Faith, and Fantasy in the New Physics of the Universe. The chapter entitled “Fashion” covers String Theory, adding his perspective. The chapter covering quantum mechanics is entitled “Faith.” Techniques of quantum electrodynamics show success, the evidence being experimental validation to accuracies rivaling relativity. Penrose added an experimental test of macro‐quantum superposition. The chapter entitled “Fantasy” deals with cosmologies beyond the big bang. Penrose highlighted “The Phenomenal Precision in the Big Bang”; of particular interest to us in thermo‐fluids was the necessarily low entropy.
Regardless of a theory's popularity, political or otherwise, we urge experimental tests.
Sometimes an experiment planned for another purpose provides the answer. A prime example was the search for residual thermal evidence of the big bang. While a Princeton physics group was proposing to test the theory, a couple of astronomers at AT&T Bell Labs, Arno Penzias and Robert Wilson, were trying to eliminate noise which was contaminating the signal in their antenna. They even removed pigeon and bat residue. Failing to eliminate the noise, they spoke with Princeton professor Robert Dicke. They published “A Measurement of Excess Antenna Temperature at 4080 Mc/s” in the Astrophysical Journal (Penzias and Wilson 1965). In 1978, Penzias and Wilson won the Nobel Prize.
Another Invitation from Feynman
Richard Feynman was an experimentalist as well as a theoretician. CalTech once assigned him to teach introductory physics. His notes are immortalized in The Feynman Lectures on Physics, Volumes 1–3 (1963). He remarked about fluid physics in volume 1, chapter 3 of his lectures:
There is a physical problem that is common to many fields, that is very old, and that has not been solved. It is not the problem of finding new fundamental particles, but something left over from a long time ago – over a hundred years. Nobody in physics has really been able to analyze it mathematically satisfactorily in spite of its importance to the sister sciences. It is the analysis of circulating or turbulent fluids.
In volume 2, Feynman went into more depth in chapter 41 entitled “The Flow of Wet Water.”
In a personal letter, Feynman admitted “The theory of turbulence (I have spent several years on it without success).” Turbulence was an area of classical physics beyond even Feynman's ability to solve. To this day, basic turbulence remains unsolved. As an added incentive, one of the seven Millennial Problems awaits solution of the Navier–Stokes equations which govern fluid flows.
Consider Feynman's challenge as an invitation to thermo‐fluids, experimental or theoretical. It is the most challenging area of classical physics. Turbulence becomes further complicated by heat transfer; yet more complicated by mass transfer; yet more by chemical reactions or combustion; yet more complicated by electromagnetic interactions. It is important for flight, for weather, for breath and blood, for life, for engines, for circulation within celestial stars. Flows of liquids, gases, and plasmas are found at the microscopic scale within living cells to the astronomic scales between galaxies.
One of our colleagues, Professor Adrian Bejan, overlaps with us in the same field and the same publisher. Bejan's Constructal Theory has brought a fresh theoretical approach to thermo‐fluid systems, to urban planning, and to appreciating design in living creatures as well as other fields.
Extra Invitations to Experiments
At the time of the first edition, two popular TV shows featured experimental scientists. NCIS featured Abby the forensic scientist and Dr. Mallard (Ducky) the coroner; not only are they scientists, they are the most well‐rounded characters. The Big Bang Theory featured five experimentalists: Bernadette (biology), Leonard (physics), Amy (neuroscience), Raj (astrophysics), and Howard (engineer and astronaut); and one theoretician, Sheldon. The one nonscientist, Penny, was the most well rounded in reality and society.
Finally, a fun yet serious invitation. Physicists dispute a “theory of measurement” concerning mysterious quantum behavior. In certain experiments, if a measurement is made the experiment behaves one way; if no measurement is made the experimental results are opposite. The contrary results are repeatable. Include in the dispute: Is a human required as observer? Can a cat be the observer? If so, would Schrödinger's cat endanger itself? An internet search of this topic results in millions of hits, so we recommend Richard Muller's Now: The Physics of Time (2016).
Since the idea of this book is to equip you, dear Reader, to plan and execute taking measurements in your experiment, this dispute is moot.
Panel 2.3 Prepublishing Your Experiment Plan
We advise publishing and presenting your experiment plan to your client before collecting data. In the same vein, Button et al. in Nature Reviews Neuroscience (2013) made the following recommendations for researchers:
Perform an a priori power calculation. [Our note: This helps estimate sample size.]
Disclose methods and findings transparently.
Preregister your study protocol and analysis plan.
Make study materials and data available.
Work collaboratively to increase power and replicate findings.
These recommendations combined with our strategies and guidelines help to achieve a credible experiment.
2.4.7 Surveys and Polls
If your type of experiment is a survey or poll, this text is also for you. The same strategies for sampling, statistics, and modeling apply. For a poll to promise any sort of accuracy, the sampling must be random and guided by good statistical practice. The principal challenges are designing nonleading questions and obtaining a representative sample. Otherwise a poll will mislead. Who has not witnessed and pained over a misleading poll?
Surveys of selective scientific audiences have a mixed history. One survey is particularly notable to us as experimentalists. In 2006, a survey of the International Astronomical Union (IAU) in Prague became memorable when the IAU demoted the planet Pluto. Pluto had been discovered and recognized as a planet in 1930. For decades teachers and textbooks across the world taught the nine planets in the solar system. Then the press in 2006 widely reported that the IAU declared that Pluto was no longer a planet. From our viewpoint, as IAU outsiders, it was a decree with huge impact on classrooms and the textbook industry.
As experimentalists, we are interested in the sampling, statistics, accuracy, and credibility of the Pluto survey. These all have relevance to us as we consider “The Nature of Experimental Work” (this chapter). We read that within the IAU there is dispute and controversy. How well did the vote represent the entire population?
Let's review details: The total population of IAU members is about 11,000. In 2012, the IAU reported 10,894 individual members from 93 countries worldwide. At the 2006 Prague conference, 2,400 members had registered. On the last day of the conference, just 424 astronomers remained and were polled on several issues. On some issues, exact numbers of the vote were reported (IAU 2006). However, the definitions