Mixed results for polls in Ontario by-elections

Two polling firms were very active during the Ontario by-election campaigns: Campaign Research and Forum Research. Their polls had a very mixed performance, with two races being called well, one having the broad strokes called accurately but not the precise numbers, and two being called poorly. That gives them a grade of 2.5 out of 5, straddling the pass/fail mark.

But before getting to how the polls did, how did ThreeHundredEight do? The By-Election Barometer was designed to forecast outcomes in general terms in by-elections where precise, reliable information is hard to come by. On that score, the Barometer extended its streak without a missed call to 18 in a row. Scarborough-Guildwod was considered Likely Liberal and stayed Liberal. Windsor-Tecumseh was considered Strong New Democrat, and it was won in a landslide. Etobicoke-Lakeshore and Ottawa South were considered PC/OLP toss-ups, and they were both close races won by the PCs and OLP, respectively. London West was considered a three-way toss-up, and the NDP took it.

The Barometer also gives an indication of how the parties met expectations. The New Democrats won both of the ridings they were considered to be at play in (and for awhile looked like they could take a third in Scarborough-Guildwood), the Liberals went 2 for 4, while the Progressive Conservatives went only 1 for 4. In the end, though, all five were Liberal seats - and generally safe ones at that.

Compared to the polling, ThreeHundredEight did well with my weighted averages (which take into account survey date, sample size, and firm track record) having a lower average error than either Forum or Campaign. The simple turnout model, however, performed poorly: it had a worse average error than either Forum or Campaign. It makes me doubt whether it is worth its salt (it didn't have much salt to begin with, though).

Now, to the polls themselves.

Etobicoke-Lakeshore was the race that was called the best last night, at least in general. Forum had a total error of 4.3 points, significantly better than Campaign's 9.9 points. ThreeHundredEight's weighted average, with an error of only 3.5 points, was the best performer. The turnout model did as poorly as Campaign, though unlike Campaign it still had Doug Holyday as the winner. While the results for Campaign were not far from their forecast, having the winner wrong is a problem. Their under-estimation of PC support puts them just on the edge of their own margin of error, and this was their biggest issue.

Turnout was roughly 39% in the riding (going by the total number of eligible voters from the 2011 election, rounding down to take into account population growth since then).

This was a big win for the Progressive Conservatives, since it gives them a seat in Toronto and a big name in caucus. But who won this riding: Doug Holyday or Tim Hudak? Considering the poor performance of the Tories in other ridings, the answer seems pretty clear. The Liberals put up a decent performance, but did lose about nine points here from the 2011 election. Drop them by nine points throughout Toronto and they are in trouble.

London West was not polled very well, as neither Forum not Campaign had the New Democrat candidate Peggy Sattler as the winner. Forum was the closest, however, with a total error of 14 points against 16.9 points for the weighted average, 17 points for the turnout model, and 22.8 points for Campaign.

But the feather in Forum's cap in London West is not a good one. The result for the NDP and PC candidates was outside the margin of error, whereas Campaign only had the New Democrats outside the margin of error out of the major parties. The turnout model did suggest that the NDP had room to grow, but the Tories should not have been boosted. All in all, a miss.

Turnout was roughly 38% in this riding.

This is a big win for the New Democrats, who have won all three of the by-elections that have taken place in southwestern Ontario since the last provincial election. They gained about 20 points, while the Liberals lost about 30. This was a big defeat for them, as Ken Coran was supposed to be a star candidate. Instead, he had the second worst Liberal performance of the night.

Ottawa South was the worst polled riding of the five, with the polls suggesting Matt Young would win easily against John Fraser. Instead, Fraser won by just under four points.

Campaign did better than Forum with a total error of 19.2 points, but they had Young ahead by seven points. The weighted average performed second best, with a total error of 21.9 points, while Forum had a total error of 26.6 points. The turnout model was the worst of all - it had the Tories ahead by 21 points!

This was a complete miss by Forum, as the results for the Liberals, PCs, and NDP were outside the margin of error. Campaign had only the Liberals outside the margin of error of the three major parties, so it was not a complete miss by them. But it was not a good performance by any stretch of the imagination.

Turnout in Ottawa South was roughly 40%, the highest of the night.

The Liberals have to be happy with this win. To many, a loss here would have represented a repudiation of the McGuinty years as well as a refusal to accept that the new government of Kathleen Wynne is a different beast. The Liberals' loss of support was relatively modest. For the Tories, this was a strong performance - but they had hopes of taking it and from that perspective it is a blow. If they can't win McGuinty's old riding when campaigning against McGuinty's memory, they may have to change their messaging.

Scarborough-Guildwood was well served by the polls, with Forum coming the closest with a total error of only 6.4 points. Campaign had a total error of 8.8 points, their best performance of the night. In fact, this was the only riding in which both Forum and Campaign did better than ThreeHundredEight's weighted average.

This was a good call by Forum in that the four major parties were called within the margin of error. Unfortunately for Campaign, however, Adam Giambrone's performance for the NDP was outside the margin of error. The turnout model suggested that the NDP could do better than Forum had envisioned, but it was over-zealous.

Turnout in this riding was roughly 36%.

A win is a win and the Liberals (and certainly Mitzie Hunter) will take it, but they did drop about 13 points in this riding. The New Democrats picked up nine, which is a strong performance for them. With the PCs also taking Etobicoke-Lakeshore, it puts the Liberals on notice that their Toronto seats are far from secure.

Windsor-Tecumseh was not well polled (the total error was actually worse than London West), but unlike the other misses it did not mislead people. The New Democrats won it easily, as the polls suggested. But that doesn't mean the polls were close.

The weighted average was the best performer, off by 17.4 points. Campaign was off by 18.7 points, while Forum was off by a total of 20.7 points. Worse, their results for the NDP and Tories were outside the margin of error. Campaign was outside of the margin of error for the NDP and Liberals. It was a poor performance by the polls, but at least no one in Windsor-Tecumseh was surprised by the results.

For that reason, perhaps, turnout was the lowest here at around 30%.

Because this win was expected by Percy Hatfield and because the riding votes NDP federally, it does not look as impressive as it should. But the Liberals lost 30 points to the NDP, and it seems that with Dwight Duncan gone the Windsor area is firmly in the NDP camp at the provincial and federal level. Unlike, say, London West or Kitchener-Waterloo, this is a riding that will almost certainly remain an NDP fortress going forward.

Overall, ThreeHundredEight's weighted average was the best performer, with a total average error of 13.9 points per riding, or 2.8 points per party (including Others). Forum Research was close behind, with a total average error of 14.4 points, or 2.9 points per party.

Campaign Research had an average error of 15.9 points or 3.2 points per party, while the turnout model was the worst performer. It had an average error of 18.8 points per riding, or 3.8 points per party.

The turnout model had improved every Forum election-eve poll they had done in the recent past, but it made them all worse here. Similarly, turnout went against the grain. It averaged 38% in the ridings retained by the government, whereas it averaged 36% in those won by the opposition. My research for yesterday's Globe article showed that turnout was, on average, quite a bit lower in by-elections won by the incumbent. It was a topsy-turvy night.

It is hard to spin these results as a good performance for the polls. It was not. While Etobicoke-Lakeshore and Scarborough-Guildwood were well-polled, 2 for 5 is a poor performance (especially in light of the 19 times out of 20 caveats that go along with margins of error). Forum did well in Etobicoke-Lakeshore and Scarborough-Guildwood, but had poor performances in London West and Windsor-Tecumseh and a very bad performance in Ottawa South. Campaign had a generally bad night, as in addition to the misses in London West, Ottawa South, and Windsor-Tecumseh, they had the Liberals winning Etobicoke-Lakeshore and were outside of the margin of error for the NDP in Scarborough-Guildwood.

But by-elections are not easy things to poll, so this shouldn't be taken as another indictment of the industry. The sample sizes were very small, and the response rates very low. Both firms used IVR technology, polling in one evening over a couple of hours. In Campaign Research's reports they mentioned how they needed to call well over 20,000 households to get a few hundred responses, a response rate of under 2%. Undoubtedly, Forum had a similar rate of response. Regional variations in behaviour and availability at the national level probably even things out more, but in a by-election those 2% might be a very different set of people than the rest of the riding. But this is a problem for the pollsters to figure out - they shouldn't put out numbers if they know them to be potentially faulty.

For my part, I'm pleased that the By-Election Barometer continues to give a good, broad-strokes overview of by-elections and that it does not rely too heavily on individual polls (for instance, it still had the Liberals favoured in Ottawa South). It will continue to operate, and hopefully its success will continue in Toronto Centre, Bourassa, and Provencher. But the turnout model may be turfed.