In this guest post, Frank Graves of EKOS Research investigates.
“LIKELY” VOTERS AND THE NDP VOTE IN THE ONTARIO ELECTION: A CAUTIONARY TALE
Frank Graves
Looking back at the Ontario election, we were very pleased to have predicted not only the victor, but also the majority. Only one other firm made this correct forecast. While we are very happy with the nearness of our final poll to the actual vote tallies for the two frontrunners, we were considerably less pleased with the performance of our Likely Voter (LV) model and our underestimate of NDP support. We were a bit over four points low on the NDP, but in a majority government this underestimate is of little practical consequence. The far more important challenges were identifying the winner, the leader of the opposition, and whether it was a majority or not; all of which we accomplished clearly and accurately. Yet in a closer race, the underestimate of NDP support could have been far more consequential and the challenge of forecasting who will vote and who won’t remains an important and unsolved puzzle.
As it turns out, the failure of the LV (LV) model, and the underestimate of the NDP are closely linked. The LV model significantly lowered NDP vote from a modest to large error and it moved the Liberal and Conservative votes from spot on to too high. Clearly, this wasn’t the intention and this error is somewhat baffling as the ingredients of the model are based on established historical patterns which show significant connections between these terms and likelihood of voting. To recap, we know that generally speaking (ceteris paribus), individuals who don’t know where there polling station is, who are younger and display lower socioeconomic status, who haven’t voted in recent elections and who don’t tell us that they are absolutely certain to vote, are less likely to vote. It is therefore extremely puzzling to see that applying these factors to estimate who will show up to vote not only wasn’t helpful, but magnified the error.
Further analysis of our final polls shows why this failure occurred and it is directly linked to the underestimate of NDP support. These could be quite separate issues, but an analysis shows they are interdependent. First, let’s look at the connection between our LV model and the final result.
The basic premise of our LV model involves assigning each respondent a “likely voter” score (maximum of eight points), based on a number of factors, such as age, past vote behaviour, knowledge of voting station, etc. Since we know that voter turnout is likely to be in the range of 50%, we isolate the (roughly) 50% of respondents who received the highest scores (giving us a “cut-off” of six points). Whether we apply the model weakly or strongly makes little difference. We find that different cut-off points make some impact but basically no model decreases error; they all magnify it. This is perplexing as we and others have shown clear linkages to these terms and voter turnout.
We then looked at the individual impacts of each of the terms of the model and none of the individual terms improved things. In part, the poor performance of the LV model is rooted in the paradoxical reverse linkages between turnout and the NDP vote. NDP vote appeared to be modestly rising in the later stages of the campaign, but we didn't make much of it because it was focused in parts of the voter spectrum that are not linked to high turnout: lower education, younger ages.
Worse, the NDP vote was much more concentrated in non-voters from 2011, they were less likely to know where their polling station was located, they were considerably less certain to vote, and they were less likely to have voted in the advance polls. A few other notable findings: the NDP were performing more strongly with households with children at home and they were more likely to be union members. The link to the deep cuts to public sector workers may have caught the attention of these groups as the campaign matured.
One other interesting feature was evident in the data: NDP supporters were much more likely to only use a cellphone. We do sample cellphones, but we were under the population values and didn't weight this group up. This was because we were burned in the 2011 Federal Election for capturing cell only respondents who were much less likely to vote and less supportive of the Conservative government. In this instance, cellphone-only households were less likely to be Liberal or Progressive Conservative supporters, but they were not less likely to vote. If we had sampled more cellphone-only households, we would have probably been closer to the actual NDP vote.
In the end, we're left conclusion that even the most cautious and empirically informed attempts at creating LV models can be dangerous because things change; sometimes quite significantly. This harkens back to the classic problem of induction and the fact that the future will often not resemble the past. We are quite certain that presidential hopeful Mitt Romney was very surprised that he didn't win, as was Frank Newport of Gallup for precisely this reason. We therefore propose a more modest and less influential LV strategy that avoids the more ambitious approach we tried here. We also think that greater effort needs to be put into ensuring full coverage of all groups (e.g., cellphone-only households and younger voters), even if these groups hadn't been critical in past exercises.
We now have a reasonable handle on the reasons behind the underrepresentation of NDP supporters and the failure of the LV model. Our solution is a lighter less ambitious LV adjustment focusing on likelihood of voting. We also think that greater efforts to represent younger voters and cellphone-only households would be a prudent strategy.
We are, however, left with one big unanswered question. What is it in the latter stages of the campaign that seemed to engage what should have been a relatively disengaged NDP base to actually show up? That question is a very interesting one to which we have no real answer at this stage. A post-election survey would help to clarify this critical question.
What we do know is that this improbable appearance on Election Day was in defiance of historical patterns of likely voting and reinforces our conclusion to put more effort into creating the best random samples and less into seeking an elusive, unified field theory of voter turnout which we increasingly believe is a chimera.
