Everything we thought we knew was wrong?

It just wouldn't be an election in Canada anymore without an unusual poll from Forum Research raising eyebrows and over-turning the weight of evidence collected by other polling firms. Last night, Forum did it again in a new poll first reported by the Montreal Gazette.

The poll gave the Liberals 40% support against 38% for the Parti Québécois and 12% for the Coalition Avenir Québec. In the larger scheme of things, those results are not too far from the latest findings from CROP and Léger. But underneath the top line results, the numbers mark a serious departure from what the two Quebec-based firms have been finding to be the case for months. Or do they? More on that below.

The vote projection has moved rather dramatically as a result of this poll being added to the aggregate. The PQ has been bumped up just 0.8 points to 37.8%, but the Liberals have jumped 2.7 points to 37.6%. Much of that came from the CAQ, which fell 1.8 points to 13.6%.

It hasn't shifted the seat projection very much, however. The PQ has slipped one seat to 68, with the Liberals at 50 seats. The ranges have changed a bit more, with the PQ dropping from 62-81 seats to 61-78 and the Liberals increasing from 36-56 to 42-60. The likely ranges still give the PQ the victory, but a majority is less assured than it was on March 5. The CAQ's range has dropped from 5-7 seats to 1-5.

Now to the Forum poll, which was conducted on the first day of the campaign via interactive voice response. On the one hand, it is good to have a different methodology in the field. Both CROP and Léger use online panels, though CROP reverted to live-callers in the 2012 election campaign. We will have to see if they do so again. On the other hand, Forum's one-night flash polls are not the best way to go about collecting a sample. There's no time for call-backs, so instead of a poll of eligible voters in Quebec it is a poll of eligible voters in Quebec who happened to be home on Wednesday night. Are they different than those Quebecers home on Thursday nights?

As mentioned, the top line results are not too unusual. Though the Liberals are certainly higher than the 34% to 35% recently recorded, perhaps the start of the campaign has boosted them. The 38% for the PQ is well in line with other results, as is the 7% for Québec Solidaire. The 12% for the CAQ is low (Forum always seems to have the CAQ lower than other firms), but it is not much different from the 15% and 16% registered in the last polls by CROP and Léger.

But then we get to the linguistic breakdowns. Forum has the PQ at 41% among francophones, low compared to CROP and Léger but not unreasonable (that is where they were in polls conducted between October and January). But Forum has the Liberals at 36%. That is an incredible number - though almost exactly where Forum has the party in their last polls in September and October of last year. But just as those results were at odds with what CROP and Léger had at the time, they are again. According to those two firms, the Liberals have been at or below 25% among this demographic for the last two months. The last time either firm had the Liberals as high as 36% was years ago.

The non-francophone score is also quite high for the Liberals: 89%. But that is not an implausible number, as CROP has had the party that high in the last year.

It would seem that these numbers from Forum are unweighted, because otherwise they make no sense. If the Liberals are at 36% among francophones (who make up roughly 80% of the population) and 89% among non-francophones (who make up the remaining 20%), they would be at roughly 47% support province wide. The PQ's score would translate into about 34% support. This is likely due to the over-sampling of Quebec City, where the Liberals are at 46% support. This would have been done to build a useable sample. But it would seem that Forum did not re-weight their reported francophone numbers to take that into account (they would have to in order to calculate the provincial numbers, however). In other words, the number for the Liberals among francophones is not weighted correctly - that is the raw number, meaning the Liberals are NOT at 36% support among francophones.

Instead, the Liberals would seem to need to be around 28% for the overall numbers to make any sense, with the PQ around 47% and the CAQ around 16%. This would put Forum's numbers more in line with what other firms are showing.

The big difference, then, comes primarily in Quebec City. The numbers in Montreal and the regions are not dissimilar from those of the other firms, but the Liberals have not been put higher than 34% in any polls by CROP and Léger since December. The margin of error for a sample of 144, which is what Forum reports for the Quebec City region, is +/- 8%. So, it is possible that the Liberals are pushing the limits of that MOE in the provincial capital.

So, perhaps the poll is not as absurd as it appears on first glance. But these are things to keep in mind going forward: Forum's reported support for francophones seems unweighted, meaning the numbers are worthless if they are not regionally weighted. Focus must thus remain only on the regional and overall numbers if we are to take anything useful from the Forum polls.