Australasian Science Magazine

By Peter Bowditch

Anomalous or unexpected results will always be a part of scientific research.

Two classic songs from my youth had the title Do It Again. One, by The Beach Boys, has become a staple on oldies’ adult rock radio stations, and the other, by Steely Dan, can be regularly heard on smooth jazz outlets. I don’t think that Brian Wilson, Walter Becker or Donald Fagen were thinking about science when they wrote these songs, but they always remind me of the importance of replication and reproducibility in the process of scientific research.

Anomalous or unexpected results will always be with us whenever scientific research is being done. It may seem strange to say that unexpected results are expected but that’s just a quirk of the English language.

The research projects that I undertook while studying cognitive psychology usually contained the dreaded “p<.05” somewhere in a discussion of the results. I know that some physicists ridicule this and other similar findings in the social sciences and medicine, and hold it up as evidence that these things aren’t real sciences, but probabilities like this in results merely indicate that we are dealing with things that are variable or difficult to measure precisely and recognise that the results might not be right.

It can sometimes be hard to explain to people that it simply means that if a large number of similar trials were conducted, we would expect this result more than 95% of the time. It is this uncertainty that forces us to repeat experiments, sometimes many times and preferably by different researchers, to see whether the probability of error holds up.

One of the problems with pseudoscience research is that the researchers are prepared to accept the first set of favourable results that come along and to claim that the theories have been proven correct.

A classic example of this was the purported confirmation by Margaret Ennis and others of the work of Jacques Benveniste which apparently showed that the effect of an active agent could survive dilution beyond the point when any active ingredient is to be left in the mixture. Ennis claimed to be highly sceptical of the idea of homeopathy, and this is used as further evidence to support that what she found must be correct because confirmation bias can be ruled out.

There were obvious flaws in methodologies used by both Benveniste and Ennis, and when better controlled and managed experiments were carried out the magic effect disappeared. Homeopaths claim that this does not show that there is any problem with the research because conditions were not replicated exactly. Scientists respond that exact replication using the faulty methods of the original research would actually prove nothing, because one of the reasons for replication is to attempt to remove possible areas of error.

In psychic research the classic method is to keep going until you find what looks like evidence for what you are trying to prove, and then stop. I was declared the person with the strongest psychic powers in a room once because I correctly predicted seven coin tosses in a row. To a researcher in the paranormal this might be all the evidence necessary to state that I had some kind of predictive power. The real fact was that if you start with about 120 people in a room and eliminate people whenever they guessed incorrectly, the most likely outcome was that someone would get six or seven correct guesses in a row. This experiment can be repeated with any sufficiently large group of people and almost always produces a similar result, but it doesn’t say anything about the psychic or paranormal powers of the last man standing.

The ability to replicate results by itself is not sufficient evidence to accept anomalous results without further question or investigation. Each attempted replication should try to remove possible ways that previous results might have been mistaken. This is particularly important when investigating things that seem to be at the limits of possibility.

Repeatability of experiments and findings has long been of interest to philosophers of science. It is essentially what David Hume was saying when discussing miracles – a miracle is something that cannot be repeated because it defies the laws of nature. Charles Broad gave this a name in 1949 when he used the term Basic Limiting Principles, essentially the laws of nature that limit what is possible.

When sceptics criticise pseudoscience research they often try to avoid a priori statements of what is possible and demand that further research be done to show that some phenomenon does or does not exist. Perhaps it would be better to not even bother to do research into the possibility of the impossible.

To return to popular music, one of the early hits for The Eagles was Take It to the Limit. We should certainly do that, but perhaps we don’t need to go any further.

Peter Bowditch is a former President of Australian Skeptics Inc. (www.skeptics.com.au).