Ontario: Projection vs. Results

The 2011 Ontario election was a tricky one. Throughout the campaign, the polls showed a neck-and-neck race between the Liberals and the Progressive Conservatives. Some of the polls showed a widening gap between the two parties in the final days, but the end result wound up being just as close as the campaign had promised it would be.

The Liberals under Dalton McGuinty won their third consecutive election, but were reduced to a minority government by a margin of one seat. Tim Hudak's Progressive Conservatives formed the Official Opposition while Andrea Horwath's New Democrats remained in third place. Both opposition parties, however, improved their standings in the legislature and all three leaders stayed on.

ThreeHundredEight projected a Liberal majority of 58 seats, with the Progressive Conservatives taking 29 and the NDP 20. The end result was 53 seats for the Liberals, 37 for the Progressive Conservatives, and 17 for the New Democrats.

This was a total error of 16 seats, or 5.3 per party.

Using the actual provincial vote result improved things only slightly, to an error of 4.7 seats per party. The Liberals were still erroneously projected to win a majority.

Whether using the projected vote or the actual vote, the riding level accuracy was 85.0%, with 91 of 107 ridings being accurately called.

The seat ranges, however, envisioned a Liberal minority as it stretched from between 51 and 62 seats. But for the PCs, the upper seat range was 35, or two seats less than the party actually won. For the New Democrats, the lower seat range was 19, or two seats more than they actually won. But the ranges did give a clearer picture of what was possible.

The vote projection under-estimated the Tories by 2.1 points and over-estimated the NDP by two points. But generally speaking, an error level of 1.6% per party was not bad at all.

The chart below shows a few quick facts about the projection at the riding level.

The average margin of victory in the ridings that were incorrectly called was 8.9%, so a relatively significant amount.

The New Democrats and Progressive Conservatives were generally well called, as between 70% and 75% of their riding level projections were within 5%. This dropped to about 59% for the Liberals, and overall 45% of ridings, or 48 out of 107, had every party at or within 5% of their actual result.

On average, Liberal riding projections were 0.4 points higher than their actual result, while NDP projections were 0.8 points higher. For the Tories, the average riding-level projection was 2.5 points lower than their actual result.

This was the real problem. The Progressive Conservatives were under-estimated in 61 of 107 ridings, and only over-estimated in 36 ridings. Ten were called exactly.

This contrasts greatly with the Liberals and NDP, who generally split evenly. The New Democrats were under-estimated in 50 ridings and over-estimated in 48, with nine being called exactly, while the Liberals were over-estimated in 53 ridings and under-estimated in 46, with eight being called exactly.

In all, the margin of error at the riding level was +/- 3.3% per party.

These are the top three ridings in terms of accuracy:

For Niagara West-Glanbrook, the Liberals and NDP were only off by one point apiece, while in Oak Ridges-Markham the projection was off by one point for the Tories, NDP, and minor parties. In Thornhill, the projection was off by one point for the NDP, the Greens, and the minor parties.

But why did the projection get 16 ridings wrong? It wasn't just the polls under-estimating the Progressive Conservatives, as the model would have still gotten 16 ridings wrong with the actual province-wide numbers.

The problem is that province-wide numbers are difficult to use in such a regionally diverse province. If we look at the results more closely, we see where things failed regionally. The bulk of the incorrect ridings were in southwestern Ontario, where the Tories were under-estimated in 17 out of 21 ridings. They were also under-estimated in every riding in northern Ontario.

For the three major parties, the riding level projections had an average error of +/- 4% or less in central Ontario, the Hamilton/Niagara region, and the GTA. But that error was more than 5% in Toronto, southwestern Ontario, and the north, where more than two-thirds of the errors took place. This demonstrates why a regional model is necessary, despite the lack of uniformity in how the pollsters divvy up the province.

This was a big reason for a lot of the errors, but there were also some specific factors at play. For example, Rocco Rossi was not a "star candidate" in Eglinton-Lawrence, or at least that bonus made that riding an inaccurate call.

Overall, however, it was not a bad projection. The 1.6% popular vote error was acceptable, as was the +/- 3.3% error at the riding level. Being off by 5.3 seats per party is less acceptable, but had the Liberals eked out their majority it would not have been as consequential an error as it turned out to be.