Bad Pharma: How Medicine is Broken, And How We Can Fix It. Ben Goldacre
see in the last bullet point, the data on adverse events is also censored: I’m reproducing the whole page here because I worry that it would otherwise be almost too bizarre for you to believe. This level of censorship isn’t an everyday phenomenon; but it illustrates the absurdity of what medicine has come to accept, in professional documents that most people wouldn’t bother to read.69
Why shouldn’t we all – doctors, patients and NICE – have access to the information on trials that regulators see? This is something I asked both Kent Woods from the MHRA, and Hans Georg Eichler, Medical Director of the European Medicines Agency, in 2010. Both, separately, gave me the same answer: people outside the agencies cannot be trusted with this information, because they might misinterpret it, either deliberately or through incompetence. Both, separately – though I guess they must chat at parties – raised the MMR vaccine scare, as the classic example of how the media can contrive a national panic on the basis of no good evidence, creating dangerous public-health problems along the way. What if they released raw safety data, and people who don’t know how to analyse it properly found imaginary patterns, and created scares that put patients off taking life-saving medication?
I accept that this is a risk, but I also believe their priorities are wrong: I think that the advantages of many eyes working on these vitally important problems are enormous, and the possibility of a few irrational scaremongers is no excuse for hiding data. Drug companies and regulators also both say that you can already get all the information you need from regulators’ websites, in summary form.
We shall now see that this is untrue.
Two: Regulators make it hard to access the data they do have
When exposed to criticism, drug companies often become indignant, and declare that they already share enough data for doctors and patients to be informed. ‘We give everything to the regulator,’ they say, ‘and you can get it from them.’ Similarly, regulators insist that all you need to do is look on their website, and you will easily find all the data you need. In reality, there is a messy game, in which doctors and academics trying to find all the data on a drug are sent around the houses, scrabbling for information that is both hard to find and fatally flawed.
Firstly, as we’ve already seen, regulators don’t have all the trials, and they don’t share all the ones that they do. Summary documents are available on the early trials used to get a drug onto the market in the first place, but only for the specific licensed uses of the drug. Even where the regulator has been given safety data for off-label uses (following the paroxetine case above) the information from these trials still isn’t made publicly available through the regulator: it simply sits quietly in the regulator’s files.
For example: duloxetine is another drug in fairly widespread use, which is usually given as an antidepressant. During a trial on its use for a completely different purpose – treating incontinence – there were apparently several suicides.70 This is important and interesting information, and the FDA holds the relevant data: it conducted a review on this issue, and came to a view on whether the risk was significant. But you cannot see any of that on the FDA website, because duloxetine never got a licence for use in treating incontinence.71 The trial data was only used by the FDA to inform its internal ruminations. This is an everyday situation.
But even when you are allowed to see trial results held by regulators, getting this information from their public websites is supremely tricky. The search functions on the FDA website are essentially broken, while the content is haphazard and badly organised, with lots missing, and too little information to enable you to work out if a trial was prone to bias by design. Once again – partly, here, through casual thoughtlessness and incompetence – it is impossible to get access to the basic information that we need. Drug companies and regulators deny this: they say that if you search their websites, everything is there. So let’s walk, briefly, through the process in all its infuriating glory. The case I will use was published three years ago in JAMA as a useful illustration of how broken the FDA site has become:72 replicating it today, in 2012, nothing has changed.
So: let’s say we want to find the results from all the trials the FDA has, on a drug called pregabalin, in which the drug is used to treat pain for diabetics whose nerves have been affected by their disease (a condition called ‘diabetic peripheral neuropathy’). You want the FDA review on this specific use, which is the PDF document containing all the trials in one big bundle. But if you search for ‘pregabalin review’, say, on the FDA website, you get over a hundred documents: none of them is clearly named, and not one of them is the FDA review document on pregabalin. If you type in the FDA application number – the unique identifier for the FDA document you’re looking for – the FDA website comes up with nothing at all.
If you’re lucky, or wise, you’ll get dropped at the Drugs@FDA page: typing ‘pregabalin’ there brings up three ‘FDA applications’. Why three? Because there are three different documents, each on a different condition that pregabalin can be used to treat. The FDA site doesn’t tell you which condition each of these three documents is for, so you have to use trial and error to try to find out. That’s not as easy as it sounds. I have the correct document for pregabalin and diabetic peripheral neuropathy right here in front of me: it’s almost four hundred pages long, but it doesn’t tell you that it’s about diabetic peripheral neuropathy until you get to here. There’s no executive summary at the beginning – in fact, there’s no title page, no contents page, no hint of what the document is even about, and it skips randomly from one sub-document to another, all scanned and bundled up in the same gigantic file.
If you’re a nerd, you might think: these files are electronic; they’re PDFs, a type of file specifically designed to make sharing electronic documents convenient. Any nerd will know that if you want to find something in an electronic document, it’s easy: you just use the ‘find’ command: type in, say, ‘peripheral neuropathy’, and your computer will find the phrase straight off. But no: unlike almost any other serious government document in the world, the PDFs from the FDA are a series of photographs of pages of text, rather than the text itself. This means you cannot search for a phrase. Instead, you have to go through it, searching for that phrase, laboriously, by eye.
I could go on. I will. There’s some kind of ‘table of contents’ on the seventeenth page, but it gets the page numbers wrong. I’ve finished now. There is simply no reason for this obfuscation and chaos. These problems aren’t caused by technical issues specific to trials, and they would hardly cost any money at all to fix. This is plainly, simply, unhelpful, and the best we can hope is that it’s driven by thoughtlessness.
That’s a tragedy, because if you can unearth this document, and decode it, you will find that it is full of terrifying gems: perfect examples of situations in which a drug company has used dodgy statistical methods to design and analyse a study, in such a way that it is predestined – from the outset – to exaggerate the benefits of the drug.
For example, in the five trials on pregabalin and pain, lots of people dropped out during the study period. This is common in medical trials, as you will shortly see, and it often happens because people have found a drug to be unhelpful, or have had bad side effects. During these trials you’re measuring pain at regular intervals. But if some people drop out, you’re left with an important question: what kind of pain score should you use for them in your results? We know, after all, that people dropping out are more likely to have done badly on the drug.
Pfizer decided to use a method called ‘Last Observation Carried Forward’, which means what you’d expect: you take the last measurement of pain severity while the patients were on the drug, from just before they dropped out, and then paste that in for all the remaining pain measures that they missed, after they stopped coming to follow-up appointments.
The FDA disapproved of this: it pointed out, quite correctly, that Pfizer’s strategy would make the drug look better than it really is. For a fairer picture, we have to assume that the drop-outs stopped taking the drug