How did the polls do in the Ontario by-elections?

Two by-elections were held in the Ontario ridings of Niagara Falls and Thornhill last night. I wrote about how the parties themselves did for The Huffington Post Canada, but let's take a look at how the polls did here.

As is usually the case, Forum Research was the only pollster in the field. The firm was a little less active than it was in the Ontario and federal by-elections of 2013, and conducted its final poll on February 11, rather than on the eve of the vote as Forum has usually done.

Forum and the Toronto Star did seem to be aware of the skepticism that had grown with Forum's by-election polls after the misses in Ottawa South and Brandon-Souris. In their reports, the Star pointed out that Forum uses a proprietary weighting formula that was shown to the Star, and that the raw data of the polls had been deposited at the University of Toronto (if you're a political science student there, take a trek down to the library and take a look at that raw data, please!).

There was no Ottawa South or Brandon-Souris type error last night in the polls. As forecasted, Wayne Gates of the NDP won in Niagara Falls and Gila Martow of the PCs won in Thornhill. The final poll in Thornhill was quite close, with all parties' results falling within the reported margin of error, while the final poll in Niagara Falls over-estimated the support of the NDP. Otherwise, the poll did fine.

A few notes on the charts below. The "With reported MOE" row shows the margin of error ranges for each of the parties when applying the margin of error that was reported (Forum rounds the margin of error off, and calculates it for the entire sample, not just decided voters). The "Actual MOE" row shows what the margin of error for each party actually was, taking into account their level of support (a party with 50% support has a higher margin of error than a party with 5%) and the number of decided voters.

In Niagara Falls, Forum over-estimated the support of the New Democrats by at least five points, and as much as twice of that. So that was a miss, but at least the winner was correctly identified.

The PCs, Liberals, and Greens all did slightly better than the poll expected, though the results were within both the reported and actual margins of error.

The poll conducted by Forum on February 5 would have been much closer. That survey had the NDP at 38% to 36% for the Tories and 19% for the Liberals.

The poll in Thornhill was much better. The results were within three points for the PCs and within 1.8 points for the Liberals, NDP, and Greens. All results fell within the reported and actual margins of error, and comfortably within them (unlike the PCs in Niagara Falls, who fell just within the edge of the margin of error).

This is not the first time Forum has had a good performance in the Toronto area. Its Toronto Centre poll in the November by-elections was the closest of the four, while the polls in Etobicoke-Lakeshore and Scarborough-Guildwood were the best of the five Ontario by-elections of August 2013. Forum has had much less success outside of the GTA. I cannot speculate why that might be.

UPDATE: A reader made an interesting suggestion of why that might be. Forum's by-election record has been generally good in races where voting patterns did not change much (i.e., the GTA) but has been very poor in ridings where those patterns changed significantly. If Forum is making assumptions to model the voting population in its polls, those assumptions may be incorrect when the electorate is shifting so greatly.

This site's By-Election Barometer continues to have a flawless record. Thornhill had been forecast to be a Likely PC win, while Niagara Falls was a Toss-Up that favoured the NDP. This stretches the barometer's record to 29 by-elections without an error.