Poll aggregation methodology

The following is a detailed description of how the polling aggregate is calculated. While this method is similar to how the vote projection is calculated in the run-up to and during an election campaign, there are some differences.

If you are looking for the methodology being employed to aggregate the polls for the 2015 federal election, please see here.

Poll aggregation

The projection model starts with the aggregation of all publicly available opinion polls. Polls are weighted by their age and sample size, as well as by the track record and past performance of the polling firm.

The weight of a poll is reduced by 35% with each passing week outside of an election campaign and each passing day once a campaign has officially begun. The 'date' of the poll is determined by the last day the poll was in the field.

The sample size weighting is determined by the margin of error that would apply to the poll, assuming a completely random sampling of the population. The margin of error for a poll of 1,000 people, for example, is +/- 3.1%. A poll with a sample of 500 people has a margin of error of +/- 4.4%. Rather than giving the poll of 500 people half the weight of the poll of 1,000 people, the smaller poll would be weighted at 70% (3.1/4.4) of the larger poll.

An analysis of a polling firm's past experience in a province or at the federal level has suggested that polling firms that were not active in a jurisdiction's previous election have a total error 1.18 times that of firms that were active in the previous election. Accordingly, polling firms with prior experience in a jurisdiction are weighted more heavily than those that have none.

Polling firms are also weighted by their track record of accuracy over the last 10 years. Their accuracy rating is determined by three factors: 1) the last poll the firm released in an election campaign, 2) their average error for all parties that earned 3% or more of the popular vote, and 3) the amount of time that has passed since the election. In order to take into account changes of methodology or improvements made over time, the performance of a polling firm in a recent election is weighted more heavily than their performance in an older election. The difficulty of each election is also taken into account: elections where the average error was lower are weighted more heavily than elections in which the error was higher. This is meant to take into consideration elections in which there were particular factors contributing to pollster error that were outside of the pollster's control. Conversely, in elections where the consensus was close to the mark a pollster has fewer excuses for higher error levels.

The accuracy rating is determined by comparing the average error, weighted by how recent the election is, of the best performing polling firm to others. For example, if the best performing firm had an average error of 1.5 points per party, a firm with an average error of three points per party would be given half the weight.

All of these ratings are combined to give each poll in the projection model a weight (no poll is ever awarded more than 66.7% of the total weight, unless there have been no other polls done recently). In short, this means that newer polls with larger sample sizes from experienced polling firms with a good accuracy record are weighted more heavily than older and smaller polls from inexperienced firms with a bad track record.

The performance of this method

This adjusted and weighted poll aggregation performs better than most individual polls and better than an unweighted and simple averaging of the last polls of a campaign. In 16 federal, provincial, and municipal elections, ThreeHundredEight.com's vote projection model has outperformed the average error of the final polls conducted by all pollsters during a campaign polls 14 times and has, on average, had an error level of 2.15 points per party compared to 2.68 points per party for the polls.