Planning and Executing Credible Experiments. Robert J. Moffat
Dr. Ioannidis then compared these “results of highly cited articles … against subsequent studies of comparable or larger sample size and similar or better controlled designs. The same analysis was also performed comparatively for matched studies that were not so highly cited.”
Although part of the same article, this collection of research studies fared better than those mentioned in our Chapter 1. “Of 49 highly cited original clinical research studies, 45 claimed that the intervention was effective. Of these, 7 (16%) were contradicted by subsequent studies, 7 others (16%) had found effects that were stronger than those of subsequent studies, 20 (44%) were replicated, and 11 (24%) remained largely unchallenged.”5
In the same year, Dr. Ioannidis published “Why Most Published Research Findings Are False,” a provocative title. Although the wording appears to encompass all fields, the examples in the article were medical experiments. In order to make his evaluations, he adopted a key metric called the “Positive Predictive Value” (PPV). From this research, Dr. Ioannidis deduced the following “corollaries about the probability that a research finding is indeed true”:
Corollary 1: The smaller the studies conducted in a scientific field, the less likely the research findings are to be true.
Corollary 2: The smaller the effect sizes in a scientific field, the less likely the research findings are to be true.
Corollary 3: The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true.
Corollary 4: The greater the flexibility in designs, definitions, outcomes, and analytical models in a scientific field, the less likely the research findings are to be true.
Corollary 5: The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.
Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.
Be aware. Beware small studies (i) and small effects (ii). Beware explaining indiscriminately (iii). Beware lack of standards (iv). Beware political favors and conflicts of interest (v). Beware fashionable science (vi). The corollaries provide fair warning for all research, and do affect the Nature of Experimental Work.
Feynman's words suggest that experiments help science to be self‐correcting. However, Ioannidis (2012) gave the warning “Why Science Is Not Necessarily Self‐Correcting” in a more recent paper concerning psychological science.
Dr. J.P.A. Ioannidis has done more for us than exposing flaws in medical science. He invites better experiments in the biological and medical sciences. He sharpens experimental discernment.
Dr. Ioannidis is not alone. For example, Simera et al. (2010) report in the European Journal of Clinical Investigation how researchers collaborated and top publishers have agreed to guidelines regarding health research studies. There are many similar groups now.
Furthermore, there is incentive to resist bad and fraudulent research. RetractionWatch.com keeps track of research that has been retracted by author, publisher, or sponsor.
Overall, the concerted effort to assure the credibility of experiments keeps us in good company.
Panel 2.2 Selected Invitations to Experimental Research, Insights from Theoreticians
The desire to better understand how our world works invites us to credible experiments. Beyond measuring, we predict what we expect to measure. Our interest may be our company's factory floor, economics and marketing, agriculture and environment, medicine, information, engineering and technology, climate, or the sciences biology, chemistry, and physics.
Richard Feynman in his lecture “There's Plenty of Room at the Bottom” (1959) invited scientists into a new field which we now call “nanoscience.” The launch of nanoscience has led to innovative products that affect our daily lives. Nanoscience, as a research field, reaches from measuring individual atoms within molecules, to manipulating atoms, to fluid nano‐arrays for medical and pharmaceutical tests and beyond.
Better Invitation than a Nobel Prize
My (RH) first experimental mentor in aerodynamics was W.S. Saric. He told me the advantage our area had over other physics fields: in thermo‐fluid physics, researchers shared their techniques. His reasoning: since Ludwig Prandtl, an early pioneer in fluids, had not won a Nobel Prize, no one in our area of classical physics would expect to win. Classic thermo‐fluid physics was crucial to so many areas of science. The camaraderie of joint effort was inviting.
Essentially every Nobel Prize in Chemistry recognizes experimental work. Likewise, essentially every Nobel Prize in Physiology or Medicine recognizes experimental tests. How about for Physics Nobel Prizes? An accounting (Quantum Coffee 2014) of the Nobel Prizes in Physics up to 2014 divided as such: theory, 30.75 prizes (28.7%); experiment, 76.25 prizes (71.3%). Nobel Prizes in Physics for technical innovations (reckoned as experiment) were 22.2%, almost as much as theory.
The Higgs mechanism, theoretically predicted in 1962, eventually culminated in the announcement of the experimental discovery of the Higgs boson in 2012 at the Large Hadron Collider (LHC). The LHC employs thousands of scientists and engineers. F. Englert and P.W. Higgs received the 2013 Nobel Prize in Physics for their theoretical work. The Higgs Nobel Prize was a rare instance where theory preceded experiment. Einstein, like most theorists to win, won the prize for explaining an experiment.
Einstein's Theory Always Invites Tests
Einstein's theory of relativity is arguably of the most precisely tested theory in science, with experimental agreement to better than the 12th decimal place. The measurements allowing such fine precision and accuracy involved a binary pulsar (Antoniadis et al. 2013). Although it is so well tested, several times a decade we read about an experiment claiming to violate or refute Einstein's theory. The experimental results which appear to refute are expertly considered and critiqued. Invariably a flaw in technique or instrumentation is discovered, further confirming Einstein's theory rather than refuting it. Confidence in Einstein's theory increases with each test. In 2018 a test beyond our galaxy was reported and confirmed.
We trust Einstein's theory as far as it has been experimentally tested, not due to its popularity.
Observations of a Popular Theoretical Physics Field
In the early 1980s, String Theory became a popular physics field. Our particular interest for this text is twofold: (i) its fashionability invites comparison with Ioannidis corollaries 5 and 6; (ii) Ioannidis evaluated medical research based on falsifiable predictions, called “PPV,” as discussed in Panel 2.1.
During the peak years of String Theory popularity, its math techniques flourished. It garnered the majority of physics funding; its proponents placed the majority of professorships. We avidly read about it.
Since String Theory is one of several competing theories, and outside our specialty, we continue to watch with interest all sides in the dispute.
String Theory notably depended on multiple spatial dimensions. E.A. Abbott's book Flatland: A Romance of Many Dimensions had already introduced us to imagining extra spatial dimensions. E. Witten was a top advocate of String Theory. A principal concern, stated by advocates and critics alike, was that String Theory lacked predictions that could be tested experimentally.
Beginning