Australasian Science: Australia's authority on science since 1938

Counting the Opinions

By Peter Bowditch

The same-sex marriage survey repeats the statistical mistakes of most opinion polls.

People who might never have heard of George Santayana would still be familiar with the quote: “Those who cannot remember the past are condemned to repeat it”. Australia has been going through a process that repeats mistakes from the past.

The 1936 US presidential election was between the incumbent Democrat Franklin D. Roosevelt and Republican Alf Landon, Governor of Kansas. Roosevelt was partway through implementing his New Deal program, something which was seen by many conservatives as ushering in a form of socialism and anathema to those who saw libertarianism, small government and individual personal freedom as the foundations of US polity.

The magazine The Literary Digest mailed out about 10 million survey forms to subscribers and others, asking them to predict who would win. The 2.3 million replies suggested that Landon would win 370 of the 531 Electoral College votes, and therefore the Presidency. Yet Roosevelt received 500 Electoral College votes and more than 60% of the popular vote.

So how did The Literary Digest get the prediction so wrong? One suggestion was that the subscriber base of the magazine consisted predominantly of conservative voters, so they were the wrong people to ask. The real problem, however, was a combination of this and also that the respondents were a self-selected sample of people motivated enough to fill in a form, put it in an envelope and go to a postbox to return it.

Another player in 1936 was advertising executive George Gallup, who thought that there was a scientific way of predicting the results of the election. Using a sample of 50,000 people he said that Roosevelt would win.

I remember having hammered into me in the first statistics courses I did at university the importance of sample selection, and how samples had to be both random and representative of the population under study. These might seem conflicting ideals, but it is possible to randomly select subjects while still ensuring that the overall sample is representative.

Large amounts of advertising expenditure and media company profits are based on television and radio ratings. These ratings are based not on counting everybody who watches a particular television program but on monitoring the behaviour of a very carefully selected sample of a couple of thousand people across the country.

Political polling can be very accurate with a small sample, but that sample is drawn from every electorate in the country. (The 1975 Australian Federal election that followed the dismissal of the Whitlam government was one of the most polarised in history. The newspaper Nation Review had a competition to predict the relative percentages in two-party preferred voting for the Senate. The winner, who got the percentages correct to the third decimal place, worked for the Gallup organisation and used the figures from an opinion poll a week before the election.)

Australia is in the midst of a massive opinion poll to answer a single question: should the law be changed to allow same-sex couples to marry? Showing that we have learnt nothing from the lessons of 1936, the survey is being conducted as a postal vote. Although everybody on the electoral roll has been sent a survey form, participation is optional so only people really motivated by the question will feel the need to respond.

Motivation to express an opinion might not even be enough. I live in a small country town with approximately half of the residents of the local government area living on farms outside the town. There appears to be only one postbox in the entire area for people to return their voting papers, and I’ve spoken to many people who only come into town about once a month to do shopping. If you don’t post a lot of letters it’s easy to forget that you need to take one with you the next time you go for groceries.

Three weeks after the first voting forms were posted out there were three estimates of the response rate. Two surveys conducted by private pollsters with sample sizes of about 1000 suggested that approximately 75% of the forms had already been returned. The Australian Bureau of Statistics said in the same week that 57.5% had been returned. The ABS figure was itself an estimate based on counting envelopes in a few boxes and extrapolating to a warehouse full of boxes; if about 25% of the votes were coming in per week then just a few days delay in postal delivery could account for the difference between the two estimates.

I’ve posted my vote, but I’ll have to wait until mid-November like everybody else to see the result (and of course the ABS analysis of the validity of the count). I hope we get value for the large amount of money the survey cost.


Peter Bowditch is a former President of Australian Skeptics Inc. (www.skeptics.com.au).