The New Democrats slipped to 22.9% (or between 21% and 25%) and between 16 and 23 seats. The Greens also dropped, to 6.5% (or between 5% and 8%). The extreme ranges no longer envision a Green seat. For the other parties, however, both the Liberals and PCs could potentially win a majority, or a minority government. Only the PCs can reasonably bank on finishing second or better.
But let's look at the Forum poll for the Toronto Star, which kicked off the polling for the 2014 Ontario provincial campaign. There are a few issues worth noting.
All of those shifts appear to be within the margin of error. However, the New Democrats have dropped in three consecutive Forum polls going back to February, when the party was at 26% support. The Liberals and Tories have been wobbling to and fro since then. A drop in Andrea Horwath's approval rating, to 36% from 40%, and an increase in her disapproval rating, to 41% from 34%, may be of some concern for her party.
Before we get to Forum's seat projection, let's look at the sample itself. I have highlighted some of the sampling issues with Forum's polling before. Most pollsters have issues with surveying younger Canadians, but none seem to have it to such an extent as Forum.
The sample
This survey of 1,845 Ontarians included 1,689 who answered Forum's voting intentions question. Whether this is the product of their weighted sample or the number of decideds (which would imply about 8% undecideds) is not explained in their report. But what the report does show is that 69% of those 1,689 respondents were 55 or older. That is a bit of a problem, as only about 36% of the electorate in Ontario is 55 or older.
Why did this happen? Only 5% of the sample is made up of voters aged 34 or younger, for a grand total of 83 respondents. The actual proportion of the electorate in this age group is about 26%.
This does not mean that the overall poll results are inappropriately skewed towards older respondents. The Forum report explains that "the data has been statistically weighted by age, region, and other variables to ensure that the sample reflects the actual population according to the latest census data." This means that the 69% sample of older voters would normally be reduced to the appropriate 36%, and the 5% of younger voters would be inflated to 26%.
But that is not exactly what happened. The report in the Star stipulates that "Forum uses a weighting formula, which has been shared with the Star, to more accurately reflect the broader electorate." We can surely trust that the demographic and polling experts at the Star approved of this formula, if they exist.
So that means that Forum is not necessarily inflating that 5% young sample to represent 26%, nor that the 69% old sample has been deflated to 36%. If Forum's "secret sauce" under-represents younger voters and over-represents older voters, we can expect the proportions to be different. What they are, we don't know (but the Star does).
Even so, that sample of 83 younger voters is a bit of a problem. The results for that demographic were 35% for the Tories, 27% for the Liberals, and 23% for the NDP. Younger voters, then, are flocking to the Tory banner. That is counter-intuitive, to say the least. No worries: a sample of 83 has a margin of error of +/- 11%. So, roughly speaking, the PCs could actually be anywhere between 24% and 46%, the Liberals between 16% and 38%, and the NDP between 12% and 34%. Knowing what we know about young voters, the Tories are probably at the lower end and the other two parties (particularly the NDP) at the higher end.
But that sort of assumption wouldn't be made when Forum applies its weighting formula. Those 83 respondents would instead be inflated to whatever Forum's formula says it should be. Let's say it was inflated three times to represent just 15% of the electorate. If that 35% for the Tories should have instead been 24%, that means the PCs may be getting a bonus of one to two points overall, due to the sampling error. Those 83 respondents are now meant to represent a sample of 249, which would normally have a margin of error of +/- 6%. If it had been a sample of 249, then the difference for the Tories due to maximum normal sampling error would have been less than a point.
Put simply, there is the potential to magnify the effects of sampling error if the sub-samples need to be re-weighted to such a great extent. In essence, it also increases the overall margin of error of the sample. This poll of 1,845 Ontarians is not as valuable as a poll of 1,845 Ontarians that had a more representative sample. That does not mean the Forum poll is wrong - merely that there is a greater chance that it could be than might otherwise have been the case.
The seat projection
Forum's seat projection for their poll grabbed a bit of attention. Despite the decent PC lead, the Liberals were projected to win 49 seats, with just 45 going to the PCs and 13 to the NDP.
Maybe I am less qualified to speak about polling than the experts who work in the field, but I certainly know a thing or two about seat projection models. And Forum's seat projection model has always puzzled me, since the methodology has never been published anywhere. When I have asked, I've been told it is a normal swing model. Outside of Ontario, Forum's numbers never look too strange (they gave the Quebec Liberals 85 seats in their final Quebec poll, rather than the actual 70, but that was due to their over-estimation of the Liberal vote by 3.5 points). But their provincial projections for Ontario have never made much sense to me, as they have always under-scored the New Democrats to a significant degree.
When I plug Forum's regional numbers into my model (which means I am using the exact same data that Forum is using, with the same regional distribution), I get 45 seats for the PCs, 41 for the Liberals, and 21 for the New Democrats. It is very difficult to fathom how Forum can get the NDP at just 13 seats with 22% support - exactly where they were in 2011 - with the PCs up just two points over that election and the Liberals down almost five. In fact, I'd have to have the NDP at around 18% support before I'd project them to have 13 seats.
To put that into context, if Forum's vote and seat numbers were exactly what happened in an election, it would rank as my model's 11th worst performance over 13 elections.
And this is not even the most egregious example of under-scoring the NDP. In a January 2013 poll, Forum gave the NDP the lead with 35% against 32% for the Tories and 27% for the Liberals. The seat projection? A tie at 40 seats apiece - for the Liberals and Tories. The NDP would win just 27 seats and finish a distant third in the seat count, all the while holding a province-wide lead in support. A poll in September 2012 put the NDP in second but 15 points ahead of the Liberals - who would win just one less seat than the NDP!
Certainly, part of this is a product of our electoral system. My current projection gives the PCs a three-point lead but puts them in a tie for seats. We have seen real-world examples of this in Manitoba and Quebec. But the electoral geography is not so greatly stacked against the PCs that they can't win a plurality of seats without a much larger lead in the vote share. As a rule of thumb going forward, I'd suggest increasing the NDP's seat share that is estimated by Forum by 50%, all at the expense of the Liberals. Or just focus on the vote intention numbers. Those of us who don't work for the Star don't quite know how Forum comes to them, but their 'secret sauce' was the most successful in the 2014 Quebec election, as well as the last election in Ontario. I leave it to you to decide how much this blurs the line between polling and modelling.
