The Failure of Risk Management. Douglas W. Hubbard
of risk management is that there is usually no experimentally verifiable evidence that the methods used improve on the assessment and mitigation of risks, especially for the softer (and much more popular) methods. If the only “evidence” is a subjective perception of success by the very managers who championed the method in the first place, then we have no reason to believe that the risk management method does not have a negative return. For a critical issue like risk management, we should require positive proof that it works—not just accept the lack of proof that it doesn't. Part of the success of any initiative is the measurable evidence of its success. It is a failure of risk management to know nothing of its own risks. It is also an avoidable risk that risk management, contrary to its purpose, fails to avoid.
2 Some parts that have been measured don't work: The experimental evidence that does exist for some aspects of risk management indicates the existence of some serious errors and biases. Because many risk management methods rely on human judgment, we should consider the research that shows how humans misperceive and systematically underestimate risks. If these problems are not identified and corrected, then they will invalidate any risk management method based even in part on human assessments. Other methods add error through arbitrary scales or the naive use of historical data. Even some of the most quantitatively rigorous methods fail to produce results that compare well with historical observations.
3 Some parts that do work aren't used: There are methods that are proven to work both in controlled laboratory settings and in the real world, but they are not used in most risk management processes. These are methods that are entirely practical in the real world and, although they may be more elaborate, are easily justified for the magnitude of the decisions risk management will influence.
In total, these failures add up to the fact that we still take unnecessary risks within risk management itself. Now it is time to measure risk management itself in a meaningful way so we can identify more precisely where risk management is broken and how to fix it.
SCOPE AND OBJECTIVES OF THIS BOOK
My objectives with this book are (1) to reach the widest possible audience of managers and analysts, (2) to give them enough information to quit using ineffective methods, and (3) to get them started on better solutions.
The first objective—reaching a wide audience—requires that I don't treat risk management myopically from the point of a given industry. There are many existing risk management texts that I consider important classics, but I see none that map the breadth of the different methods and the problems and advantages of each. There are financial risk analysis texts written specifically for financial analysts and economists. There are engineering and environmental risk texts for engineers and scientists. There are multiple risk management methods written for managers of software projects, computer security, or disaster recovery. Many of these sources seem to talk about risk management as if their methods comprised the entire subject. None seems entirely aware of the others.
The wide audience objective also means that I can't write just about the latest disaster. A reader picking up the first edition of this book in 2009 may think the risk I'm talking about is a financial risk. If I had written this just after the Fukushima Daiichi nuclear disaster of 2011 or more recent events, then risk might have meant something very different. But risk is not selective in that way and the best methods are not specific to one category of risks. Thinking about risks means thinking about events that have not yet occurred, not just last year's news.
Finally, reaching a wide audience requires that I don't just write another esoteric text on quantitative methods for a small community of experts. Of those, there are already some excellent sources that I will not attempt to reproduce. A couple of slightly technical issues will be discussed, but only enough to introduce the important concepts. So, I will spend very little time on well-developed methods in actuarial science or quality control in engineering. The focus will be more on where there are numerous competing methods and the highest levels of management such as ERM.
The last two objectives—to get managers to quit using ineffectual methods and start them on a better path—are also satisfied by a just-technical-enough approach to the problem. This book won't make most managers masters of more quantitative and scientific methods of risk management. I merely want to convince them to make a radical change in direction from the methods they are most likely using now.
To accomplish these objectives, the remainder of this book is divided along the lines implied by the title:
Part One: An Introduction to the Crisis: This first chapter introduced the problem and its seriousness. Chapter 2 outlines the diversity of approaches to assess and mitigate risks and discusses how managers rate their own firms in these areas. Chapter 3 examines how we should evaluate risk management methods. Chapter 4 will show a simple “straw man” that can be the basis for developing a fully quantitative model. (This will also provide a way to imagine an alternative to current risk management methods as we go through a long and detailed criticism of them.)
Part Two: Why It's Broken: After an introduction to four basic schools of thought about risk management, we will discuss the confusing differences in basic terminology among different areas of risk management. Then we will introduce several sources of fundamental errors in popular methods that remain unaddressed. We will list several fallacies that keep some from adopting better methods. Finally, this part of the book will outline some significant problems with even the most quantitative methods being used.
Part Three: How to Fix It: This final part will introduce methods for addressing each of the previously discussed sources of error in risk management methods. We will build on the basic straw man model introduced in chapter 4. We will discuss the basic concepts behind better methods, including how to think about probabilities and how to introduce scientific methods and measurements into risk management. Finally, we will talk about some of the issues involved in creating a culture in organizations and governments that would facilitate and incentivize better risk management.
Throughout this book, I will offer those who require more hands-on examples sample spreadsheets on this book's website at www.howtomeasureanything.com/riskmanagement. Those who prefer the 10,000-foot view can still get a good idea of the issues without feeling dragged down by some technical details, whereas those who prefer to get more information can get specific example calculations. The website will also give all readers access to evolving risks, new ideas and a community of other professionals interested in commenting on those.
See this book's website at www.howtomeasureanything.com/riskmanagement for detailed examples from the book, discussion groups, and up-to-date news on risk management.
NOTES
1 1. My use of placebo effect requires a qualification. The placebo effect in medicine is the tendency among patients to experience both subjective and, in some cases, objectively observable improvements in health after receiving treatment that should be inert. This is a purely psychological effect but the improvements could be in objectively measurable ways—such as reducing blood pressure or cholesterol. However, when I refer to a placebo effect, I mean that there literally is no improvement other than the subjective impression of an improvement.
2 2. Capt. A. C. Haynes, “United 232: Coping with the ‘One-in-a-Billion’ Loss of All Flight Controls,” Accident Prevention 48, June 1991.