Australasian Science: Australia's authority on science since 1938

Most Medical Research is Wrong?

By Michael Cook

An analysis of significant medical research papers has concluded that “most research findings are false for most research designs and for most fields”.

Most of us have set pragmatism as our default position on bioethics. If it works, why not use it? If human embryonic stem cells are reported to be effective, for instance, what harm can there possibly be in using them? In fact, it may be immoral not to use them after the incredible progress reported in Nature!

But in an era of science by press release, pragmatists should know how reliable such reports are. And respected studies into the credibility of all medical research – not just on stem cells – suggest that claims of incredible advances are precisely that: incredible. In fact, according to a leading medical statistician, Greek academic John Ioannides, “most claimed research findings are false”.

Dr Ioannides is not a crank or an enemy of science. On the contrary, his work has been published in leading journals and his claims are widely accepted among his colleagues. His ground-breaking 2005 paper in PLoS Medicine is the most downloaded in the journal’s history. Every year he receives hundreds of invitations to speak at conferences. Doug Altman, the director of Oxford University’s Centre for Statistics in Medicine, told Atlantic Monthly: “You can question some of the details of John’s calculations, but it’s hard to argue that the essential ideas aren’t absolutely correct”.

Ioannides’ claims are largely statistical and thus require much brain cudgelling for laymen. But his conclusions ought to rattle anyone: that “most research findings are false for most research designs and for most fields” and “claimed research findings may often be simply accurate measures of the prevailing bias”.

Why is this? There are a number of interlocking reasons.

Many studies are too small to be reliable. The best ones involve several thousand subjects but many studies, especially in genetics, are based on fewer than a hundred. Many studies are badly designed or are hard to compare with other studies of similar data.

Prejudice plays a role as well – although not necessarily ideological or financial. Scientists who are committed to a theory are less likely to find contradictory evidence. “Many otherwise seemingly independent, university-based studies may be conducted for no other reason than to give physicians and researchers qualifications for promotion or tenure,” wrote Ioannides in his 2005 PLoS article. “Prestigious investigators may suppress via the peer review process the appearance and dissemination of findings that refute their findings, thus condemning their field to perpetuate false dogma.”

Finally: “The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true”. Ioannides attributes this counter-intuitive effect to cut-throat competition to publish exciting research first. “This may explain why we occasionally see major excitement followed rapidly by severe disappointments in fields that draw wide attention,” he says.

Isn’t this relevant to far-reaching claims made for embryonic stem cells?

Even more discouraging for medical researchers is that the gold-standard of medical research, double-blind randomised trials, are not altogether reliable either. In another 2005 paper published in the Journal of the American Medical Association, Ioannides examined 49 of the top science papers of the previous 13 years. They had appeared in the best journals and had been cited extensively, yet between one-third and one-half of them were unreliable because they were later found to be wrong or exaggerated.

Although Ioannides’ ideas are widely accepted, some researchers fear that they might be misinterpreted and used to debunk science or to promote shonky alternative therapies. He responds that truth is the best medicine: “The scientific enterprise is probably the most fantastic achievement in human history, but that doesn’t mean we have a right to overstate what we’re accomplishing”.

Sound advice for the inner pragmatist!

Michael Cook is editor of BioEdge, an internet bioethics newsletter.