Because the polls did well, the vote projection model did well. In fact, ThreeHundredEight.com's vote projection had a smaller cumulative error than any one pollster. The total error of the vote projection was 3.5 points across the four major parties. Next lowest was Forum Research, which had a total error of 5.6 points. The performance of the polls by Angus Reid and EKOS Research were also quite strong, with a cumulative error of 6.5 and 6.6 points, respectively. Léger's final poll had a total error of 8.6 points, while Ipsos Reid's poll (out of the field six days before the vote) had a total error of 13.6 points among likely voters.
The seat projection model performed well for the Liberals, giving them 69 seats (or between 60 and 78) when the party actually won 70. For the PQ, CAQ, and QS, however, the 95% confidence interval ranges were required to capture the eventual results. The PQ was projected to take as few as 29 seats, and they won 30. The CAQ was projected to win as many as 23 seats, and they won 22. These errors occurred because of the CAQ's performance around Montreal at the expense of the PQ. Of the 10 riding-level errors where the ranges did not identify the correct winner, nine of them were seats expected to go to the PQ that actually went to the CAQ. And seven of them were in the greater Montreal area.
This wasn't the fault of the polls, exactly. The regional numbers, as an aggregate, were actually quite good. Rather, it was that the CAQ won these Montreal-area seats by the skin of their teeth. A projection model can falter if a party manages to win a lot of close races.
Overall, the right call was made in 82.4% of ridings, or 103 of 125. In those 22 errors, the confidence in the call averaged just 57.6% - so most of them were genuine toss-ups. Nine of them, actually, were called with 50% or 52% confidence, and more than half of them with 59% confidence or less. The worst call was in Masson, which was projected to go PQ with a 73% chance. That was the only case of the confidence being over 69% and being wrong.
The likely winners through the projection ranges were identified in 92% of ridings, or 115 of 125. Considering the scope for error at the local level, I think that is a very positive result.
The Liberals only greatly out-performed the polls on the island of Montreal, where it won them just one more seat than expected. They took 58.5% of the vote here, compared to the 52.1% that the polls suggested. The PQ took 17% instead of 19.7%, while QS took 12% (tying their 2012 performance) instead of 12.8%.
The vote projection in the suburbs around Montreal was very, very close. The Liberals took 35.5% of the vote here, compared to the projected 34.3%. The PQ took 28.8%, compared to the expected 30.5%. And the CAQ took 26.5%, rather than 26.3%.
Quebec City was also quite close, with all parties projected to within 1.1 points. The Liberals won the region with 37.9% of the vote to 35.3% for the CAQ.
And in the rest of the province, the Liberals, PQ, and QS were projected to within 1.2 points, and the CAQ to within 2.1 points. The Liberals won here as well, with 36.9% to 29.7% for the PQ.
More specifically in the 'regions', the PQ won in eastern Quebec with 39.1% of the vote to 37.3% for the Liberals and just 14.5% for the CAQ. In central Quebec, the Liberals had 38.2% support to 30.4% for the CAQ and 23.6% for the PQ. And in western Quebec, the Liberals had 35.4% support to 30.5% for the PQ and 25.5% for the CAQ.
The regional breakdowns show that only the PQ was over-estimated across the board: by 2.7 points on the island of Montreal, by 1.7 points in the suburbs, by 1.1 points in Quebec City, and by 1.2 points in the regions. This explains their over-estimation in the seat projection. The Liberals, meanwhile, were over-estimated in Quebec City and the regions but under-estimated around Montreal. The CAQ was over-estimated in Montreal, but under-estimated in Quebec City and the regions. Even so, we're only talking about a handful of points - a spectacular performance for the polls.
As an aggregate, of course. Individually, there were mixed performances. Forum did very well in the Quebec City and Rest of Quebec regions, averaging 1.4 points of error per party or less. It averaged 2.9 points per party of error in the Montreal area, but that is still quite good considering the margin of error for smaller sample sizes.
Léger was within 1.6 points per party in Quebec City and the regions as well, but was 3.8 points off per party around Montreal.
There was no breakdown of likely voters by region in the Ipsos poll, so it is hard to gauge. But the poll of eligibile voters did well in the regions of Quebec (2.9 points per party), but less well in the rest of the province (between 5.4 and 6.8 points per party). EKOS and Angus Reid both repeated the errors in the Montreal region that other pollsters did. GOTV operations may have been a factor here.
With all the tight races, how would the model have done if the polls for the four regions defined by the model had been even more accurate?
The Liberals would have still been called to within one seat, and the third seat for QS would have been within the likely ranges. But again the PQ and CAQ results would have fallen just outside the likely ranges by one seat for the PQ and two seats for the CAQ.
This should not be shocking - there was a 20.5% chance (or about one in five) of the CAQ having its result fall above the high projection band. But there was only a 7.5% chance that the PQ's result would fall below the low projection. That an incumbent government under-performed to such a great extent says more about how motivated the PQ's supporters were than it does about the projection model. It also shows how well the CAQ did - under normal circumstances, they should not have been able to win all those close races.
The six ridings in which the winner would not have been identified by the likely ranges were Borduas, Chambly, Iberville, Masson, Mirabel, and Saint-Hyacinthe. It is no coincidence that these were all CAQ victories in which the PQ was favoured. It is also no coincidence that they were all actual pick-ups by the CAQ in the same region of the province. There was a serious and disproportionate CAQ gain in these regions that went over-and-above what should normally be expected to happen.
This marks the second consecutive election in which the polls do a good job of tracking a race, making all the hard work put into this site worthwhile. The seat projection model performed decently with the polls, and very well with the actual results. In my view, the difference between the results and the model tells us something about what happened last night, rather than about the errors of the model itself. If humans behaved in a way that was uniform and easy to predict, models like mine would be right in 99% of cases. It is where it is wrong that we see real politics at play.
