People often ask if they can trust the polls – but what really matters is to understand how polls work.
The latest Radio New Zealand/Reid Research poll puts Labour on 34.3%, and National on 32.5% – more or less on par with other recent polls, making allowance for margins of error.
The collaboration between RNZ and Reid gains credibility by the publication of a good explainer by RNZ’s Nik Dirga on how polling works generally and on Reid’s methodology. It’s important to read political opinion polls critically (not “trustingly”), so the explainer is helpful. Polling companies do their best to get representative samples (which is not easy these days) and they strive for best estimates of public opinion. But there’s always a statistical margin of error, so we shouldn’t make direct inferences from the survey sample to the whole population of voters. Polls are slightly blurry snapshots of the recent past – not forecasts or predictions of future election results.
Moreover, there’s a step from the survey results to the public reporting of the results. Media organisations make choices about how newsworthy a poll is – and how to frame it. Making it into the lead item on the 6pm news, for instance, with terms such as “shock” or “bombshell” or (my favourite) “whopping” isn’t helpful from a scientific point of view. But, if a media organisation has paid a large sum for the survey, then the incentive is to generate audience engagement by presenting it in a sensational manner. Meanwhile a market research firm gets major brand exposure.
A scientifically conducted opinion poll is worth reporting, but does it really deserve to lead the news? Is one poll all that important?
It’s reported that each Reid poll conducts 1,000 online interviews of eligible New Zealand voters aged 18 or over, chosen on demographic criteria from a pool of 40,000 volunteers. The participants don’t get paid.
What’s never disclosed is how much the research companies are paid by the media companies. That, I suppose, is commercially sensitive. But what do you think?
NB: This is not a scientific poll, of course. It’s just for feedback.
How accurate have the polls been in New Zealand? Can they influence how people vote? Do people always respond truthfully to surveyors?
Some past misuses of polls.
For most of 2023, media reports on polls were blithely assuming (or wishfully thinking) that NZ First would not make it over the 5% threshold. They translated poll results into seats in the House without including NZ First, so their tea-leaf readings turned out to be pointless – and also arguably biased against that party. NZ First always stood a chance of getting back into parliament that year – and they did come back, as I’d warned they could in March 2023:
And have you ever noticed how the same poll result can be framed quite differently by different media outlets?
For instance, does 44% support for a proposal, versus 32 percent opposed to it mean there’s “strong” support for it? Or should one say that – in that particular survey – more people supported than opposed it? Or (if we count those who weren’t sure) did most people not support it? Or should the results simply be set out in a table so that the reader makes up their own mind:
44% in favour
32% opposed
24% unsure
An editor in a newsroom (who probably didn’t study statistics) might frame the story with a biased headline which may be all a person reads.
But what about the “accuracy” of the polls?
Polls in the US (where the industry is much larger) have repeatedly underestimated support for Donald Trump.
Pre-election polls in New Zealand have generally been pretty close. The Research Assoc. NZ accepts, however, that the polls immediately prior to the 2020 NZ election underestimated “the gap between the two main parties’ level of support”. Indeed, the 2020 pre-election polls were significantly inaccurate – overestimating National’s support and underestimating Labour’s. RANZ has attributed that to “the very high number of early voters”. “By the time those last pre-election polls were in field, a significant number of respondents had already voted. Asking them who they would vote for may not be the same as asking who they did vote for.” [Italics added.] The advance voting period may have had an effect and this needs to be taken into account when interviewing participants. But the polls were just as inaccurate before the advance voting had commenced, so I don’t buy RANZ’s explanation. Or is it an excuse?
RANZ hasn’t taken account of the unexpectedly high turnout in 2020. Pollsters aim for samples that represent the population of eligible voters. But not all eligible voters actually vote. As there was an influx of people who normally don’t vote, the polls’ samples may not have mapped as well onto the final voting population, on that occasion. The sub-population of people “who only sometimes vote” is not randomly distributed through the whole eligible population. They’re probably also people who wouldn’t normally participate in opinion polls, and their political leanings might be more to the left. If turnout swings unexpectedly high or low, that could change the demographic make-up of that election’s population of voters. And 2020 suggests that high turnout may have put opinion polls significantly out of line with the election results.
Do the polls influence election results?
Opinion polls can have self-fulfilling or “bandwagon” effects on people’s voting behaviour – though not huge or one-directional effects. Our opinions aren’t scientifically measurable “things” akin to dormant physical objects in space. Surveys can have feedback effects on a surveyed population, and people don’t just read them passively. Moreover, anyone can critically evaluate the credibility of polls, although this may be shaped by their prior political opinions, rather than knowledge of survey methodology.
As always, it’s complicated. There’s no A-causes-B theory that allows us to predict whether, or how, or by how much political opinion polls influence voter behaviour – and hence we can’t predict how they may affect an election outcome. But, intuitively, there must be some effects, even though we may not like to admit that they influence our own choices at the ballot-box. In as much as they do influence voters, the effects could go in different directions for different people.
For instance, if polls show that one candidate is way ahead, then that may inspire me to vote for that person (to join the bandwagon), or to vote for another candidate for whom I have sympathy (to back the underdog), or not to bother voting at all (as it seems to be a forgone conclusion).
In an MMP election, a party that’s persistently below the 5% threshold and isn’t likely to win at least one electorate seat may get caught in a self-fulfilling prophecy (if voters treat the polls as predictions). Voters who are reluctant to “waste” their votes on a party that looked unlikely (in opinion polls) to win any seat or seats at all may go for an alternative, thus helping to ensure that that party isn’t successful.
In multi-party systems like New Zealand, many people think about “who will work with whom” in government, and the polls may affect their choices on that second-party basis. For instance, they may reason that a vote for one party means effectively voting to help put another party into government as well – which may or may not be seen as a good thing.
There are, then, positive and negative views about the opinion polls.
They can be seen as informative for voters, and hence a useful part of the democratic electoral process. Also, they give the parties vital feedback about their performance (or their perceived performance) in between elections.
They can also be seen as an unwelcome interference in the democratic process, especially when they’re framed in the media in sensationalist or biased ways. Perhaps people should be left alone to make up their minds about which candidate or party best represents them, rather than view an election as a race to a finish-line charted by polling averages.
New Zealand bans the publication of political polls on the election day itself, and some countries have longer blackout periods. In a free society, however, the state shouldn’t prohibit research firms from doing surveys or media from reporting the results. There’s a risk that an unscrupulous actor could publish a false poll result in an effort to shift choices or suppress turnout, and that’s one reason why there’s oversight of the research industry.
We should read the polls and the media critically at all times, and not take what we read “on trust”.
Now let’s visit the “the far side” of how people respond to surveys.
In 2021, a YouGov survey asked 1,224 adult Americans if they could beat a grizzly bear in a fight – unarmed. Six percent (or roughly 73 respondents) said they could. A lion was reckoned to be beatable by 8 percent. Go them! And there was no significant gender difference: men and women said yes at about the same rate.
Only two percent of Britons said the same, so could we put this down to excessive self-confidence among a small minority of Americans? Well, it could also mean that more Americans than Britons hate pollsters and lie to them for kicks. Or perhaps some Americans (bless them!) have the zaniest sense of humour.
Sure, there probably are quite a few Americans (God only knows how many!) who do delude themselves that they can defeat a grizzly bear with their bare hands. But, even if taking a robo-call, the desire for social approval or resistance may kick in – and some people just like to impress or to lie to surveyors, perhaps because they’ve had a drink and want a laugh or because they feel annoyed. (Yeah, you idiot, of course I can. Lol!)
So if four percent of Americans said they “believe shape-shifting reptilian people control our world by taking on human form and gaining power”, then we’d want to account for those who were too tired or confused to answer correctly or were just taking the piss. If 23 percent of American millennials in 2016 told a surveyor they’d prefer a giant meteor to extinguish human life above either a Clinton or a Trump presidency, it’s probably because survey respondents often just express how they feel.
As Gregory Bateson put it: don’t mistake the map for the territory.
Further reading:
See The Lizardman’s Constant is 4% if you want some laughs about polls.
Why it’s so hard to gauge support for QAnon… from which we learn that conspiracy theorists are probably much fewer in number than their visibility in the media might have led us to believe.
Dawson, S. (2024). Poll wars: Perceptions of poll credibility and voting behaviour. The International Journal of Press/Politics, 29(1), 206-226. https://journals.sagepub.com/doi/pdf/10.1177/19401612221087181
I’m reserving my opinion on the above readers’ poll for paid subscribers only…
Should media disclose the fee pay for the poll results?
Given the public-interest character of a political opinion poll, I argue that the fee paid for it should ideally be transparent, especially when it’s a publicly-funded broadcaster. The cost of the survey may have influenced the way it’s reported. But I understand that, for reasons of market competition, research companies would not want to disclose how much they charge.
Two things I look at with polls are what do the results look like when expressed as a +/- 95% confidence interval band rather than a single number, and what is the trend. Labour and National jump around a bit but their 95% confidence bands usually overlap.
As a former American and long-time observer of that culture, I fear the “go get ‘em” self-talk means that that 6 to 8 percent really do think - or want to believe - they can beat grizzlies/lions in a fight. We’re seeing that grandiosity play out in their politics now.