Australasian Science: Australia's authority on science since 1938

Scientists Don’t Turn a Blind Eye to Bias

Scientific journals should insist on more robust experimental processes, according to biologists who reviewed nearly 900,000 experiments.

The team from The Australian National University found that non-blind experiments – where scientists knew which samples they were recording – averaged a 27% stronger result than blind trials. However, their review suggests that less than one in four experiments used blind data recording.

“We found that non-blind papers tended to exaggerate differences between the experimental group and the control group,” said lead researcher Dr Luke Holman.“For example, a non-blind trial of a new drug might conclude that it is way more effective than a placebo, when in fact the drug’s true effect is rather modest, simply because the researchers’ expectations biased the results.”

The research, published in PLOS Biology, comes at a time when experimental processes are under the microscope following increased levels of retractions and a review of peer review procedures by some journals.

The team used automated “data mining” to analyse nearly 900,000 papers from the PubMed life sciences database. They also compared 83 pairs of evolutionary biology papers on similar topics in which the data were collected blind in only one of each pair.

The team found that non-blind studies rejected the null hypothesis more strongly. “Non-blind studies more confidently concluded that differences between treatment and control groups were real, and not just due to chance variation.”

The researchers believe that journals should insist on blind trials more strongly, perhaps by making prominent statements to authors and peer reviewers about the necessity of using blind trials.

Holman suggests that better training is part of the solution. “Many researchers are unaware that their expectations can introduce such strong bias, and so they don’t feel the need to work blind,” he said.