Business Experiments with R. B. D. McCullough
the placebo is administered. The reason that lack of blinding is a problem in this situation is that polio, when it is mild and not severe, can seem like a flu or other diseases. Physicians (who generally believe in the power of vaccines), when diagnosing a child with flu‐like symptoms, may be subconsciously biased to diagnose polio more often in the unvaccinated children than in the vaccinated children.
A formal review of this trial published in the leading statistical journal characterized the design as “stupid and futile” and the results of the trial as “worthless” (Brownlee, 1955).
To remedy these deficiencies in the first trial, a second trial focused only on children whose parents gave consent for vaccination. These children were randomly assigned to treatment and control groups, both of whom received an injection. The former received the vaccine, and the latter received a placebo saline injection. The observant reader will have noted already that there is no longer a sample selection problem. The results are presented in Table 1.4.
Table 1.4 Results of second Salk vaccine field trial.
| Group | Size | Rate per 100 000 |
|---|---|---|
| Treatment | 200 000 | 28 |
| Control | 200 000 | 71 |
| No consent | 350 000 | 46 |
Comparing Tables 1.3 and 1.4 is instructive. Note that the incidence for vaccinated children is roughly the same, 25 vs. 28, as is the incidence for children who did not get consent, 44 vs. 46. The incidence for the “control” groups has changed markedly, and it did not go down, but it went up! Apparently the children of parents who give permission are more susceptible to polio than the children of parents who do not give permission. Thus, the first experiment was biased against the vaccine: the treatment group that got the vaccine was more susceptible to polio, while the control group that did not get the vaccine was less susceptible to polio. No one involved in the trial foresaw this possibility, though an analyst trained in the design of experiments would have been able to guard against the possibility and eliminate it, even if she couldn't foresee it! How? By randomization! We will have much more to say about randomization in subsequent chapters.
The variable “parents' propensity to give consent” is a lurking variable that is correlated with the outcome (the child getting the disease). Several mechanisms have been suggested for this correlation. For example, less educated persons are less likely to give consent, less educated persons are likely to have less income, persons with less income are less likely to live in sterile environments, and children who live in less sterile environments are more likely to have robust immune systems that offer more protection against polio.
Regardless of the true mechanism, if treatment and control had been assigned randomly, then the effects of the lurking variable would have been spread out between the two groups instead of having too many consenters in the experimental group. Thus, randomization protects against the effect of lurking variables, even when we don't know what variables we need protection against! Remember, lurking variables are not to be confused with confounding variables; again, see the “Learning More” section for details on this point.
A postscript to the experiments is in order. The success of the trials was formally announced in 1955, and widespread vaccination began. However, a problem with the injectable vaccine quickly arose. There were six different manufacturers of the vaccine, and not all of them followed Salk's instructions for preparing the vaccine. Hundreds of cases of paralysis were reported, including cases when the paralysis began in the arm that received the injection! The program was suspended in 1955. In 1960 an even better oral vaccine was introduced and put into widespread use. This is why, when examining the data, you might not see a large drop after 1955.
Exercises
1 1.3.1 List the problems with the first Salk trial.
2 1.3.2 How did the second Salk trial overcome the problems in the first Salk trial?
3 1.3.3 Why were the sample sizes so large for the Salk trials?
4 1.3.4 Find an example of another failed medical experiment. Be sure to articulate why it failed.
5 1.3.5 Plot the number of cases of polio in the United States by year; the data are in polio.csv . The eradication of polio is obvious as time increases. What is noticeable about the period before the introduction of the polio vaccine?
1.4 What Is a Business Experiment?
We have seen that when using observational data to answer a business question, the answer we get can depend on which variables are included in the analysis, how relationships are modeled, and a host of other decisions. These types of analyses are difficult to rely on, easy to argue about, and hard to do well. Consequently, their value for informing business decisions is limited. None of these criticisms apply to business experiments.
Some persons are under the mistaken belief that an experiment is just “trying something” to see “whether or not it works,” and this approach very often leads to unusable results. For example, a large bank wanted to improve its customer service, so it designated some of its branches as “laboratories” where many ideas were implemented. So many things were changed in haphazard fashion across so many branches that it was impossible to determine which of the ideas was responsible for improved results. This was a very expensive “experiment” by one of the major banks, and it produced nothing useful – it was a complete waste of resources.
Sometimes a simple experiment can point the direction toward millions of dollars as happened at Intuit. And this was after a formal usability study gave a decidedly negative recommendation about the idea! As recounted by Thomke (2020, chapter 2),
[An] engineer noticed that about 50 percent of prospective customers tried the company's small business product 20 minutes before they had to make payroll. The problem was that all payroll companies took hours or even days to approve new customers before the first employee could be paid. Wouldn't potential customers be very pleased if they could make payroll before the long approval process was done? To make sure there was a genuine need, the engineer and product manager ran a usability study. The result: none of the twenty participants were interested in a fast payroll solution. But instead of shelving the idea, Intuit modified its webpage within 24 hours and ran a simple experiment that offered two versions of the software – one with the option to click on “pay employees first” and another one with “do set‐up first.” (When users clicked on “pay employees first” option, they got a message that the feature wasn't ready.) Contrary to the usability test results, the experiment revealed that 58 percent of new users picked the faster payroll option. Ultimately, the feature became hugely popular, lifted the software's conversion rate by 14 percent and generated millions of additional revenue.
An experiment is a statistical test by which a hypothesis is subjected to data produced according to a specific procedure in which some variable thought to affect the output is deliberately manipulated. A business experiment is merely an experiment whose purpose is to inform a business decision. By contrast, some disciplines, e.g. medicine, psychology, or biology, develop theories and then use experiments to test the theory; not so for us. Each of these disciplines has its specific theories and subjects, and therefore experimental methods need to be adapted to each discipline. For example,