By-election post-mortem

The four federal by-elections last night played out as ThreeHundredEight forecast, with the Liberals retaining Bourassa and Toronto Centre, the Conservatives retaining Provencher, and Brandon-Souris being a toss-up riding that leaned slightly towards the incumbent Tories. So slightly, in fact, that the Liberals were less than 400 votes short of winning it.

By-Election Barometer Record

This makes 22 by-elections without a wrong call by ThreeHundredEight, going back to June 2012. The Barometer will be put to the test again tonight in Carbonear-Harbour Grace in Newfoundland and Labrador. The model suggests the riding is leaning Liberal, though the PCs do have a slim chance of holding on to it. However, the forecast is for the Liberals under new leader Dwight Ball to take it by a narrow margin.

The results in Brandon-Souris, which were the most at-odds with the polls by Forum Research, showed the mettle of the By-Election Barometer. The forecast was that it was a toss-up between the Conservatives and Liberals, with the Tories considered to have a 52% chance of winning to 48% for the Liberals. In the end, the Tories won it by a single percentage point.

This was similar to the by-election this summer in Ottawa South, where the polls suggested the Tories were well-placed to steal former premier Dalton McGuinty's riding away from the Liberals. The model still considered it a toss-up that leaned slightly towards the Liberals, which turned out to be the case. It demonstrates the need to take into account the fundamentals in a riding, particularly when riding-specific polls are showing numbers that are wildly out of step with historical performances.

I assessed the performance of the polls for The Globe and Mail, and the results are not great for Forum Research. They missed Brandon-Souris by a wide margin and Provencher was off by a considerable degree as well. The polls were better in Toronto Centre and Bourassa, however. This record matches their performance in the Ontario by-elections this summer, where I gave them a score of 2.5 out of 5. Last night, they got 2.5 out of 4, for picking the right winner in Provencher but missing the respective levels of support for the Tories and Liberals.

Because of this very mixed record, I employed confidence intervals far wider than the reported margins of error of these polls. This turned out to be a good idea, but only captured some of the error. Of the 20 calls Forum Research made (one for each party + others in the four ridings), 60% fell within the 67% confidence interval and 85% fell within the 95% confidence interval. Clearly, these wide ranges will need to be stretched even further, but that is the usefulness of these extra data points.

The performance of the confidence intervals, based solely on Forum's polling, compared to the results can be seen in the chart above.

The misses at the 95% confidence interval were primarily in Brandon-Souris. The results for the Liberals and Conservatives were so off that a +/- 10-point confidence interval still did not capture the result. The Greens in Provencher also fell outside of the 95% confidence interval.

At the 67% confidence interval, only the Others in Toronto Centre and the Conservatives in Bourassa were wide of the mark. In Brandon-Souris, the Liberals and Conservatives fell outside the interval, while all parties fell outside of it in Provencher. In short, even compared to Forum's past errors in by-elections the ones in Brandon-Souris and Provencher were extraordinary.

Note: I inquired with Forum about their methods for Brandon-Souris in relation to the reports of multiple calls being received by some residents. I was told by Lorne Bozinoff that they used the same random dialing in all ridings, and that no panel of phone numbers was used to dial the same people over and over again. I was also told that if a number was called in consecutive nights, that person's results were dropped from the sample and that all three of their final polls (Forum also polled on Nov. 23, but did not publish the results which were similar to the poll of Nov. 24) were independent samples.