Pollsters themselves carry some of the blame for the growing cleavage between them and the public on matters of trust, due to the corner-cutting methodologies and unimpressive disclosure standards of some of them. But the media has played a role in shaking that trust, by reporting numbers in a manner that leave readers perplexed, confused, and ready to give up on polls entirely.
Kickstarter Update: Amazing, we've passed the 2/3rds funding mark with 135 backers for the ebook project. We still have 14 days to go! Let's keep the pledges coming in. I'll have some news about the project next week!
After the Léger/Le Devoir poll came out on Saturday, CTV picked up the story with the headline "Liberals lead in new poll; support for sovereignty waning". Their interpretation of the trendline in support for sovereignty was erroneous, as we will see, but it clearly implied that the Parti Québécois was not doing very well. It was an interpretation repeated elsewhere, and the headline was easily shared on social media. Narrative established.
After the CROP/La Presse poll came out on Wednesday, CTV was back on the story with the headline "Poll shows growing support for the Parti Quebecois". They accordingly had a picture of a happy Pauline Marois to counter the sad picture of her they had in their previous story. Narrative contradicted.
Let's deal with the first problem in how these two polls were interpreted, and one that is the most problematic since it was the most easily avoidable.
Léger reported that support for sovereignty stood at 33%, with 51% against and 16% undecided. CROP reported that support for sovereignty stood at 44%, with 56% against.
Notice anything different with those two sets of numbers? No, it is not the 11-point spread between support for sovereignty in the two polls. It is that Léger included the undecideds in their report, and CROP did not. CROP was following standard reporting practice, or at least what has been the standard for years in Quebec. Léger opted not to portion out the undecideds, something which they almost always do. It is odd that they chose not to this time, and probably should have known that as a result they would cause an unnecessary firestorm. One article even went so far as to suggest CROP was doing something unusual in excluding the undecideds.
When CTV reported that support for sovereignty was waning, it was this 33% number that they were focusing on. Yes, if support among decided voters was at 33% that would certainly be a sign that support for independence was on the wane. But if reported normally with the exclusion of undecideds, that 33% would have turned into 39%. That is well within the norm, and even at the higher end of the usual 35% to 40% support that Quebec sovereignty has registered in recent years. Worse, that 39% is one point higher than the last time Léger reported on this issue in May 2013. The headline attached to the poll was not just wrong in its interpretation, it was factually wrong, at least if you're comparing apples to apples.
Both of these polls are online surveys and both firms are correct to say that a probabilistic margin of error cannot be applied. That doesn't mean, though, that the polls don't have a margin of error of some sort. Ipsos-Reid, for example, applies a 'margin of credibility interval' to their online polls, a practice that probably captures correctly how to measure these things but which, it must be said, still raises some eyebrows in some corners of the industry. Nevertheless, these polls do have a margin of error of some kind, and if we apply the standard margin of error that would apply to a probabilistic sample of this size (Ipsos's credibility interval is usually comparable to a probabilistic margin of error), we see that the two latest polls from CROP and Léger overlap at around 41% to 42% support for sovereignty.
(If, as Michel David of Le Devoir contends, the sample was skewed and support should instead have been 37% in the CROP poll, then the overlap is even greater, at between 36% and 40% support.)
Update (Dec. 14): David published a correction notice today, noting that CROP did indeed weigh their decided sample correctly. Also, an editorial in the Montreal Gazette repeats the error of comparing the 33% number in Léger's poll to the 44% in CROP's, and then goes so far as to give their readers a lesson in statistics.
The chart above shows just how much in common the CROP and Léger polls have had on this question, especially when you consider the amount of error that has to apply to these polls. And the idea that support for sovereignty is on the uptick is certainly consistent with these trendlines. The likelihood that this is a blip, and that CROP will report a drop in support next time, is also strong - and the trendlines will still be in agreement with one another. Broadly speaking, things are remaining stable on this front and both Léger and CROP are telling the same general story.
Now to the question of whether these polls are contradictory in terms of voting intentions. Léger reported 37% for the PLQ and 32% for the PQ, a widening gap from their previous poll. CROP reported a 35% tie between the two parties, a tightening gap. How can this be?!
It be. The change in support in these two polls from their previous survey are within any sort of reasonable margin of error (if not a probabilistic one). So for it to be reported that the PQ was dropping in support in the Léger poll or that the PQ has experienced a "considerable change in support" in the CROP poll is not entirely accurate. If the parties were seeing their support change by four or five points, then yes, there would be something there. A wobble within three points is not worth a headline.
For the Liberals, support of between 37% and 38% falls within the error band of CROP's polling going all the way back to April. For the PQ, a band of between 32% and 33% falls within the last four polls done by the firm. There is not much here to go on in terms of a trend - the PQ and Liberals have been wobbling back and forth since at least September. The only real trend worth considering is the gain of the PQ between May and October.
It should be pointed out as well that results that fall on the outside edges of a normal margin of error are not as likely as results that fall closer to the actual reported number. When looking at the bands above, then, the emphasis should not be on the extremes (i.e. 36% to 37% is probably a better range for the Liberals in Léger's polling than the wider 34% to 39%).
To go on about how these two polls are contradictory, how one is showing a wide lead for the Liberals and failing support for the PQ and their option while the other shows the PQ making huge strides and sovereignty just around the corner, is to make it seem like there is a contradiction and thus that the polls cannot be trusted, when that is not the case whatsoever. Léger and CROP did their polling on different days and the two firms use different methodologies (each do their polls online, but they have different panels and recruiting methods), also contributing to some of the minor differences.
That is all that readers really need to know to explain the contradiction, and they can be told with confidence that things remain relatively stable and close between the Liberals and PQ (particularly since the PQ can pull more seats out of fewer votes due to their advantage among francophones) and that support for sovereignty is probably holding at about the same levels that it has wobbled within for years. Léger and CROP are good pollsters that do good work and are transparent about their methods and samples, particularly with their clients. There is no need to portray their findings in this dismissive way. Save the scoffs for polling firms that deserve it.
