Pratte argues that polls quasi-systematically under-estimate centre-right parties, most recently seen in last week's election in the United Kingdom. He also cites the provincial election in Alberta, the 2012 election in Quebec, as well as the Scottish referendum and the most recent vote in Israel.
He posits two explanations for this problem: conservative parties are supported by older people, who are less apt to express their views to strangers on the telephone; and right-wing views are portrayed unfavourably by journalists, artists, and intellectuals, who are important players in the public sphere. People don't want to publicly go against the grain.
We could add to this theory that if centre-right parties are disproportionately supported by older voters, they should disproportionately benefit at the ballot box, since older people vote in bigger numbers.
It is not a bad argument. Anecdotally, it certainly seems right. It feels right. But is it right?
I've looked at federal and provincial elections over the last 10 years to find out. I've compared each party's average performance in the last week of polls to the actual election result. I've included all major conservative parties (in addition to the PCs and federal Conservatives, I've included the ADQ/CAQ, the Saskatchewan Party, Wildrose, and the B.C. Liberals).
And since Pratte included them in his list, I have also included the Quebec Liberal Party. In the context of Quebec politics, the PLQ is arguably a centre-right party, and the 'shy Tory' effect Pratte cited has often been used to explain the Liberals' sometimes surprising electoral performance in Quebec.
The verdict? While there does seem to be a generalized under-estimation of conservative parties in the polls, the effect has been exaggerated and, in too many cases to make it a rule, has not happened at all.
In fact, the average error for conservative parties has been just 1.6 percentage points. That isn't nothing, certainly. That is particularly the case when we compare this performance to that of the largest non-conservative party, which polls have over-estimated by an average of just 0.2 points (in other words, negligibly).
Perhaps most importantly, in 14 of the 24 elections I looked at there was no real effect on the outcome whatsoever due to any under-estimation of centre-right support. It didn't change the winner or the order of the major parties. In another two elections, it only changed who finished second and who finished third. In only three cases did it flip expectations of which party would actually form government, and in just two did it turn a minority victory into a majority one.
This means that in five of 24 elections, or just 21% of cases, did an under-estimation of centre-right parties in the polls have a major, lasting impact on an electoral outcome.
In 20 of the above 30 cases, a conservative party was over- or under-estimated by more than a point, meaning that in one-third of cases the polls were close enough that we can't seriously identify a systemic problem with the methodology.
But still, in 20 of 30 cases the conservative party was under-estimated by the polls. That means that if you're placing a bet, you'll be more right than wrong if you guess that the conservative party in the election will beat its polls. That still means, though, that you could be wrong one-third of the time.
In many cases, assuming that the polls will under-estimate the conservative party, as Pratte suggests journalists should do more often in the future, will lead you astray.
Let's say you assume that the polls will under-estimate the main conservative party by about four percentage points (Pratte cites, with numbers, the PLQ in 2012 and PCs in 2015 in Alberta, where the error was about that large). Here are some of the errors you would have made:
- Progressive Conservatives victories, rather than defeats, in Ontario in 2011 and 2014, in Manitoba in 2011, and in New Brunswick in 2014.
- The Official Opposition role, rather than third place, going to the Progressive Conservatives in Nova Scotia in 2009.
- Conservative majorities in the 2006 and 2008 federal elections, rather than minorities.
- Landslide victory, rather than a closer result, for the B.C. Liberals in 2005 and 2009.
- Minorities, rather than majorities, for the Liberals in 2007 in Ontario and Quebec.
- Would hardly have been helpful in Alberta in 2012, since the race was between two conservative parties.
Even if you assumed the under-estimation would be more modest, you'd still make many of these errors. Enough to make up for the successes? Really, it's a wash based on an assumption that is only marginally more right than it is wrong. Complicating matters is that some pollsters already apply weighting to their numbers to try to take this into account, so for some polls you may be doubling the effect.
So where does that leave us? These sorts of discrepancies are already taken into account in ThreeHundredEight.com's projection ranges.
More broadly, it is certainly something which should be considered when looking at the polls in the final stages of a campaign. The conservative or centre-right party might indeed be under-estimated, and there is more of a chance of that happening than the main centrist or centre-left party beating its polls. But it won't happen every single time, and it isn't a rule.
If you assumed the federal Conservatives would beat their polls, you would have been right in 2008 and 2011, but wrong in 2006. If you assumed the Quebec Liberals would beat their polls (as virtually everyone does in Quebec), you would have been right in 2012 and only marginally so in 2007 and 2014, but you would have been wrong in 2008.
A more responsible approach for journalists would be to remind readers that this can happen, rather than assure them that it will. If polls and human behaviour were so predictable, no one would ever get it wrong.
