Measuring Liberal polling volatility

A few weeks ago, I was having a debate with Darrell Bricker of Ipsos-Reid on Twitter concerning the high degree of variation between poll results for the federal Liberals. Bricker felt that we were mostly looking at the fluctuations caused by using different polling methodologies and the difference between good and bad polls. I felt that there was something going on beyond the methodological differences that was causing such great volatility.

I investigated this question in today's article for The Globe and Mail, which can be found here. Please read it, as it goes into greater detail than I will here, where I'm mostly just going to show some of my work.

I identified the current polling period for the Liberals, stretching back to Justin Trudeau's leadership campaign launch on Oct. 2, as the most volatile since the last election. I calculated this by splitting up the last year-and-a-half into blocks of polls of similar size and calculating the standard deviation from the mean in the results for each party. The result of the calculation was that this current period of polling has shown the most volatility for the Liberals than any party has experienced:

You can also see from this chart that the standard deviation for the Liberals has been higher than for the Conservatives and New Democrats in almost every period. This is probably due to the lack of a permanent leader of the Liberal Party: respondents might be answering with a generic Liberal Party in mind, one led by Bob Rae, or one led by any of the candidates for the leadership. That introduces a degree of uncertainty that doesn't exist for the other parties. However, if that was entirely the case, we should expect the same sort of volatility for the NDP in the run-up to their leadership convention in March.

Another thing I looked at was the margin of error. Perhaps this volatility is much ado about nothing, and we're just looking at statistical wobbling. That could be, but too many of the polls' margin of error taken since the beginning of October fall outside of the average over that time.

This chart to the left shows how much the polls have differed, but also how many of them fall outside of each other's margin of error. The EKOS, Léger, Abacus, and Forum polls taken at the end of November and in December are virtually exclusive from one another in terms of their MOE bands (assuming a random sample for argument's sake, of course).

Another factor is methodology. Are live-callers returning different results than polls conducted online or by IVR? Now we might be on to something:

There has been a systemic difference in Liberal results between live-callers, IVR, and online polls. Telephone surveys with live-callers have the highest Liberal results, with online being the lowest. But the differences recorded in this latest period are well outside of the average variation between the methodologies.

As is usually the case, there appears to be no single factor that is causing such volatility in Liberal polling numbers. Methodology is definitely one factor, as well as the introduction of the leadership race as a wild-card and the events that have occurred over the last three months (by-election, Alberta gaffes, etc.).

It is interesting to me, and hopefully to others, to delve more deeply into the poll numbers from time to time - particularly when something unusual is happening. This degree of volatility makes it difficult to determine exactly where the Liberals are right now, but it can safely be assumed that these trends of volatility are likely to continue straight through to April's convention.