We Humans and the Intelligent Machines. Jörg Dräger
Whoever lives closest gets to attend. In theory. In 2012, when the city of Potsdam surveyed how effective the local catchment areas had been, the data for almost 7,000 primary school children revealed that 35 percent of them did not attend the nearest school, and 21 percent did not even attend the second nearest.14
How people react to complexity becomes evident in everyday situations and even in simple decisions. A long shelf in the supermarket, packed full of chocolate in many variations: white, whole milk, dark, with nuts, fruit, cookie chunks, as a bar or box of pralines, inexpensive or marketed as an exclusive brand. For customers, such an extensive assortment means making a choice. An experiment by psychologists Sheena Iyengar and Mark Lepper from Stanford University has proven how difficult this can be: One group of participants could choose from six chocolates, another had 30 varieties to select from. You might think that the larger the selection, the better. Far from it. The participants who had to choose between 30 options took longer, found the decision more difficult and were less satisfied in the end.15
It is true that, in principle, decisions become better the more information they are based on. After a certain point, however, this rule no longer applies. The quality of decision-making begins to deteriorate with further input because humans’ ability to process information is limited. Our cognitive capacities become exhausted, we feel overwhelmed.16 As complexity increases, so does the probability that a decision will not be made at all.17 An obvious result in the supermarket would be that the customer does not select anything at all and goes without chocolate that day.
Such moments of feeling overwhelmed occur mostly when several options are available that differ in various ways.18 Applied to the chocolate example, the customer has to consider filling, size, shape, price, cocoa content and brand. Usually people then use mental shortcuts and rules of thumb to simplify such complex tasks.19 In the supermarket, some information is ignored, other characteristics are used to limit the selection. If someone does not like nuts, chocolate containing peanuts, for example, is immediately eliminated. In addition, people reduce complexity by abstracting various qualities. In such cases, brand-name chocolate is preferred, since various desirable traits are associated with it.
Catchment areas for primary schools, however, do not allow for non-decisions or mental shortcuts. The length and safety of the route to school cannot simply be ignored, neither can the schools’ use of resources or their social mix. In order to achieve optimal allocation, the public authorities need to run through different scenarios to identify their impact on the student population as a whole. When done manually, this takes considerable time and is prone to error (see Chapter 10).
If a decision has to take many options and characteristics into account, we humans quickly reach the limits of our cognitive capacities. Software applications can, however, provide us with meaningful support, especially when we have to examine complex situations in painstaking detail or when we are tempted to simply ignore them.
No glorification
The quality of human decision-making is unsettling and impressive at the same time. Unsettling because empirical research shows that even experts in their field sometimes decide poorly, incorrectly, inconsistently or not at all. Impressive because, despite these obstacles, the overall results are improving. Today, radiologists examine many times more images than their predecessors had to cope with 20 years ago, and they diagnose them more reliably.
People set themselves goals and pursue them. And they build tools so they can achieve them. Algorithms are such tools. They can help people compensate for their own shortcomings and gain new room to maneuver. In radiology, for example, more support from machines can give humans more time to reflect. As Michael Forsting of Essen’s University Hospital says: “New technology must help us to improve – to retrieve more information from scans, to avoid errors typically made in radiology due to fatigue or tedium, to better recognize rare conditions.”20 Intelligent machines will bring the greatest social benefit not when they merely save time or human resources, but when they improve quality.
Good algorithms are needed to achieve key societal goals such as better health and more equitable education. But even those algorithms, it turns out, make mistakes: They can draw wrong conclusions or discriminate against social groups. In judging their weaknesses, however, we should not overly glorify our own abilities. We are not perfect either, we cannot do some things as well as we think we can. The interaction of humans and machines must be designed to ensure that their respective strengths are utilized in a way that reduces their respective weaknesses.
4Algorithms make mistakes
“Computers are useless. They can only give you answers.” 1
Pablo Picasso (1881–1973)
“The computer says no.” It is one of the running gags in the popular British television show Little Britain. The idea is always the same: A customer approaches Carol, an office worker, with a request and is rejected. Sometimes the scene takes place in a travel agency, sometimes in a doctor’s office, sometimes in a bank. No matter what the customer wants, whether to book a vacation, make an appointment or open an account, “The computer says no” is inevitably the answer after Carol types in the relevant query.
She is so grumpy that the problem is completely obvious: It is not the computer, but Carol’s foul mood. Perhaps she lacks the skill to fulfil the customer’s request; she is, in any event, completely unwilling to be helpful. To prevent this from becoming an issue, she blames the computer. As if to say “Sorry, nothing I can do,” every request or complaint bounces off her, no matter how convincing or affecting it might be.
The situation that the British series parodied at the beginning of the 2000s can still be found in today’s world, since, together with incompetent, unthinking or unethical users, algorithmic systems like Carol’s computer sometimes produce unwanted, unfair and inappropriate results.
The following six stories from very different areas illustrate when, where and why algorithms can be wrong. And they show how serious the consequences can be. They also give an initial idea of what we have to do or stop doing in order ensure algorithms actually serve society.
System error: Algorithms fail to do the job they are assigned to
Louise Kennedy was actually a bit nervous.2 Emigrating from Ireland to Australia was a big step for the veterinarian. She suspected that everything would not fall into her lap right away. However, she had not expected that the highest hurdle for the native speaker with two university degrees would, of all things, be an English-language test. She scored 74 points on the oral part, 79 were required. She was refused permanent residence. Who would not think of “Computer says no”?
The Irishwoman had indeed failed – because of voice recognition technology. The computer-based test is used by the Australian immigration authorities to assess oral speaking ability. Foreigners who want to live in Australia have to repeat sentences and retell a story. An algorithm then analyzes their ability to speak.
Alice Xu, originally from China, attempted to pass the test as well. She studied in Australia and speaks English fluently, but the algorithm refused to recognize her abilities, too. She scored a paltry 41 points on her oral examination. Xu did not want to give up so easily and hired a coach, who helped her pass on her second attempt with the maximum number of points. How do you improve your oral language skills so markedly in such a short time?
Her coach Clive Liebmann explains the leap in performance, revealing the absurdity of how the software works: “So I encourage students to exaggerate tonation in an over-the-top way and that means less making sense with grammar and vocabulary and instead focusing more on what computers are good at, which is measuring musical elements like pitch, volume and speed.”3 If pitch, volume and speed are correct, the test takers could, in extreme cases, talk utter nonsense as long