Business Experiments with R. B. D. McCullough
models, subjects that are often physical entities that respond predictably, and prediction methods that have small errors. In business, well‐specified models are an exception, subjects are often human (who can respond in two different ways to the same stimulus!), and prediction methods have large errors. Therefore, it is usually the case that engineering methods for experimental design will not be applicable in a business setting, and vice versa.
EXAMPLES OF EXPERIMENTS
Example I Slow payment of invoices creates numerous problems for companies, not the least of which is a decrease in cash flow and the attendant need to incur short‐term debt to cover the shortfall. Decreasing customers' payment times has obvious advantages, but how to achieve this goal is not always clear. How long after the customer receives the goods or services should the invoice be sent? If it isn't paid on time, should another invoice be sent? Or a formal letter? Or should would a phone call be more effective? One company had been in the habit of just sending repeat invoices every month and decided to take action. One set of late‐paying customers was sent a formal letter asking for payment of the late debt. Another set of customers was called. The times of these letters and calls were varied across several trials. At the end of the experiment, the conclusion was that a phone call 10 days after the due date was most effective. The overall time from initial billing to final payment was reduced from 110 to 75 days.
Example II A large, multi‐office financial services corporation wanted to improve the process by which its customers apply for credit. The immediate problem was that 60% of the applications had to be sent back for reprocessing, usually due to incomplete information (e.g. the applicant didn't write down his age or made a transposition error when writing his phone number). Not only is this expensive, but also it greatly adds to the overall processing time, which leads to many potential customers dropping out of the application process entirely. After a brainstorming session, the project team tasked with addressing this problem decided to focus on three variables: the type of application, how detailed are the instructions, and whether examples were provided. Two additional factors thought to affect the reprocessing rate are type of application (loan or lease) and region (Midwest vs. Northeast). Thus five factors with two levels each were used in the experiment (it doesn't matter which one is called level 1 or level 2) are presented in Table 1.5
Table 1.5 Factors and levels for financial services example.
| Factor | Level 1 | Level 2 |
|---|---|---|
| (A) Application type | Loan | Lease |
| (B) Region | Midwest | Northeast |
| (C) Instructions | Current instructions | Instructions with more detail |
| (D) Example | Current example | Examples with more detail |
| (E) Negative example | None (current) | Example of what not to do |
Only two of the five factors were found to affect the reprocessing rate, C and D. The application form was redesigned, and the completeness rate increased to 95%, virtually eliminating the need for reprocessing. The number of completed applications increased dramatically, since customers were no longer dropping out. All the persons who had been tasked with reprocessing were reassigned to new tasks, increasing productivity without increasing the payroll.
Example III Progressive Insurance observed that when its policyholders hired a lawyer to settle a claim, settlement time went up from 90 days to 6 months, and the payout to the policyholder went down by $100. The costs to Progressive increased by $1600 due to the need to engage lawyers for these cases. Clearly, policyholders (and Progressive) would be better served if lawyers were not needlessly involved in the process. To achieve this goal, the project team focused on the dependent variable: percentage of claimants who hired an attorney within 60 days of the accident, which had been about 36%. Brainstorming produced 59 ideas for reducing this percentage; excluding ideas that were not “practical, fast, or cost‐free” culled the number to 19. This number finally was reduced to 13, which were tested via designed experiments. When all was said and done, the percentage was reduced by eight points, with each one‐point drop representing six million dollars in savings and better service to policyholders.
One of the more surprising innovations as a result of this experiment was that Progressive began paying out more in claims! If a person's car is totaled in an accident and the insurance company insists on paying book value rather than replacement value, what is the person likely to do? Hire a lawyer! In the experiment, districts that paid more in claims had a five‐point drop in attorney involvement. The decrease in legal fees more than made up for the increase in payments to policyholders.
Example IV A company that sold telecommunications equipment to large corporations contemplated changing its customer management system to a desk‐based account manager (DBAM) system. These account managers would not work from the field, but solely from the office, making use of the telephone and video calls. This would save on travel time, increase efficiency, and, hopefully, lead to greater profit. A small number of field account managers were provided with the DBAM and trained in its use. The accounts of these managers were the experimental group. A carefully matched set of accounts from other managers constituted the control group. (We will discuss matching in Chapter 5.)
Various measures on customer satisfaction and employee satisfaction were taken on each group before and after the experiment. Costs and revenues associated with each group were measured as well. The key performance indicator (KPI) for the experiment was cost‐to‐revenue ratio. The experimental group had a KPI that was 6% lower than the control group; this implies that the DBAM was more profitable than the existing method. The ancillary measures showed that employee satisfaction did not decrease and customer satisfaction actually increased. The primary reason for the increased customer satisfaction was that it was easier for customers to contact their representatives, who were no longer unavailable while traveling. The company rolled out the DBAM to all its field agents and increased its profits while increasing customer satisfaction.
Example V Anheuser‐Busch, a beer company, wanted to determine how much money to spend on advertising. The sample of 15 marketing areas was divided into three groups: (i) 50% increase, (ii) no change, and (iii) 25% decrease in advertising expenditure over a 12‐month period. At the end of the experiment, group i achieved a 7% increase in sales, group ii had no change, and group iii had a 14% increase! A follow‐up experiment produced the same result, something that no one ever expected: decreasing advertising produced an increase in sales. This led the firm to conclude that they had supersaturated the market with advertising, and indeed the firm substantially reduced advertising without hurting sales in other markets.
1.4.1 Four Steps of an Experiment
From the previous examples, we can list the four steps of an experiment:
1 Randomly divide the subjects (e.g. customers) into groups. The researcher does not allow the subjects to pick the group – that would be a form of self‐selection that makes the data observational rather than experimental. Moreover, this division is made randomly – the researcher doesn't assign the groups for that, too, would be a form of selection that would render the data observational.
2 Expose each group to a different treatment. The researcher does not