Australasian Science: Australia's authority on science since 1938

Things Change. Get Used to It

By Peter Bowditch

How concerned should we be that only 39% of psychology research can be replicated?

In the June edition of Australasian Science I wrote about the reproducibility of scientific studies. I was mainly concerned that there were studies in pseudoscience where replication did not indicate the presence of any effects at all, and generally this is because the original studies or experiments were conducted without proper controls or procedures. The fact that much of this “research” doesn’t stand up to closer investigation is generally ignored by pseudoscientists, although they are very quick to point out that much of what is published in real scientific journals also fails the replication test.

There was much glee in woowoo world earlier this year when it was suggested that 50% of the content of medical journals may either be incorrect or out of date. This is no surprise to people who understand how the science of medicine advances.

As an example, it is perfectly reasonable to assume that papers about effective treatments for bacterial diseases were largely reduced to the status of historical relics after the discovery of antibiotics. Similarly, imaging techniques like PET and MRI made much of what was known about the treatment of physical conditions obsolete, and the changes that might result from increasing knowledge about the human genome and neuroscience will send a lot of what we now know into the dustbin.

Science is like that. It is a work in progress, we don’t know everything, and if we did science would stop.

More relevant to the issue of reproducibility is a study recently described (not “published”) in Science. The study looked at 100 papers published in second-tier psychology journals and found that on average only 39% of the results could be replicated. This finding united the pseudoscientists and those who claim that psychology is not a science.

The investigators’ findings can be summed up in this quote: “Generally evidence was weaker on replication. The stronger the evidence was to begin with, however, including a larger effect size, the more likely the results were reproduced”.

The second sentence is hardly news, as you would expect stronger effects from better evidence to give a result closer to what is happening in reality.

My initial reaction was surprise that the figure was as high as 39%. The study looked at papers in the areas of cognitive and social psychology, with cognitive studies being more replicable than social psychology research. This is not really surprising because the two areas look at different parts of the human experience.

Cognitive psychology is largely about measurement to infer internal processes that cannot be directly measured or in many cases even described by the subject. This would lead to the expectation of a somewhat lower variability between test subjects, both in a single study and over a period of time.

Social psychology, on the other hand, is much more about observation, motivation, personality and subjective experiences that can be described by the subject. We would expect a high variability across a range of subjects, and even for the same subjects at varying times.

People aren’t like electrons or photons or atoms of elements or many of the other things dealt with in the so-called “hard” sciences. Every person is unique in the cells that make up their body and the experiences and knowledge that make up their personality, and this even applies to identical twins. I am not the same person that I was 10 years ago because events in those 10 years have reshaped both what and how I think. When people talk to me about IQ testing I like to point out that the most recent time I was tested showed that I had dropped five points from when I was tested at high school. This doesn’t mean that I’m less smart now, because the denominator in the equation is a lot larger.

It is this variability, both inter- and intra-person, that makes any form of psychological study difficult to replicate. Even if the same subject group is used for the second occasion and everything else is kept the same you would expect the results to be different, and this is even more likely with studies in social psychology. None of this, however, invalidates the idea of research in the social sciences.

One aspect that needs to be considered in any examination of replication is publication bias. Journals want to publish material that is new, exciting and different to what has gone before. A paper that says “We have exactly replicated the findings of paper X as published in an earlier addition of journal Y” is hardly going to excite journal editors and push it into the next edition, but that is another topic for another day.

Peter Bowditch is a former President of Australian Skeptics Inc. (www.skeptics.com.au).