ThreeHundredEight uses a proportional swing model as its base, and that model is then tweaked and adjusted according to several factors: incumbency, place in cabinet, and the quality of a candidate. By-elections and riding polls are also included, as is the influence of prominent independent candidates and any particularly unique characteristic of a riding (i.e., the presence of Elizabeth May in Saanich - Gulf Islands in the last election).
After the campaign was over, I went back and checked on how the application of these "factors" worked out, and whether they needed to be changed. The factors were determined by using the results of the previous three elections, so the 2011 election provided another opportunity to add to the sample.
Factors are applied to the swing model. To use the cabinet minister factor as an example, it was found that being named a new cabinet minister improved a candidates performance in an election by a factor of 1.05 over and above what the provincial/regional swing would usually give. For example, if the Conservatives increased their performance in a province by a factor of 10% (i.e., they go from 40% to 44%) a newly named cabinet minister in that province would have his or her support increased by 15% (i.e., they go from 40% to 46%).
So, let's go through the factors and how they did. We'll start with new cabinet ministers, and I will explain in detail what the chart is showing.
The chart shows that the factor applied to new cabinet ministers for the 2011 election (based on the calculations for previous elections) was 1.05. It then shows that the factor in the 2011 election was actually 1.03. In other words, new cabinet ministers did 3% better than a proportion swing would have given them.
Averaging out the pre-2011 sample with the 2011 sample by size gives us a new factor of 1.04 for new cabinet ministers.
This was not a particularly successful factor in the 2011 election, as it only increased the projection model's accuracy in 43% of cases. However, there does seem to be a cabinet minister factor at work as being named a cabinet minister had a positive effect on a candidate's performance in 57% of cases (i.e., they did better than the proportional swing alone).
The "star candidate" factor was applied to any candidate deemed to be a particularly good candidate, either through renown or previous experience. Often, the parties themselves deemed their candidates to be "star" candidates. Granted, this is a subjective factor to apply but in many cases it is a necessary one.
The star candidate factor increased the accuracy of the projection in 82% of cases, and had a positive effect on a candidate's performance in 91% of cases.
Some star candidates did hugely better than the proportional swing would have had them. Roméo Saganash in Abitibi - Baie-James - Nunavik - Eeyou did 56% better than proportional swing, while Larry Smith in Lac-Saint-Louis did 59% better.
Now to incumbency. This was handled in several ways. First, if there was no incumbent a negative penalty was assigned to the candidate representing the incumbent party. Secondly, the incumbency factor was based on whether a party was losing or gaining in a particular province compared to the last election. This is because an incumbent was found to be able to do better than other candidates when a party is sinking, but at the same time would usually do worse than other candidates when a party was gaining. It would appear that an incumbent retains his or her vote when a party is doing badly, but usually doesn't soar very much when the party is doing well. In other words, they do a better job of holding on to their vote from one election to the next, no matter how the party is doing in the rest of the province.
The effect of by-elections was difficult to gauge. The practice of using both the proportional swing from the previous general election and the results of the by-election by a proportion of 50/50 was effective in 50% of cases. Not using the by-election at all would have had about the same level of accuracy. But some ridings were simply unpredictable no matter what was used, so until something better is discovered I will be continuing with the 50/50 split.
The 2011 election had a lot of riding polls for Quebec, much fewer elsewhere. I applied the riding polls to the tune of 25% in each case, with the remaining 75% being the adjusted proportional swing. Of the riding polls (and by my count 34 ridings had publicly released poll results), the last one in each riding called the winner in 59% of cases.
In 62% of cases, including the riding poll increased the accuracy of the projection for the parties projected to place 1st and 2nd in 62% of cases, and it helped project the winner in 65% of cases. On the other hand, the projection without the riding polls would have projected the winner in 68% of cases where riding polls were available. The riding polls made the model call the winner when it otherwise would have been wrong in three cases - but they overturned correct projections in four cases.
For that reason, I will be very careful in the application of riding polls in the future. I will include them when a riding has a very unique characteristic (a prominent independent, for example). But they don't seem to add much that the adjusted proportional swing model can't already do, and it is difficult to keep track of them in terms of their date (which in some cases wasn't even reported). But it wasn't the age of a poll which determined its accuracy, of the riding polls which incorrectly called the winner, at least half of them were conducted during the last week of the campaign.
In terms of who did the best (not counting riding polls that were superseded by other riding polls taken later in the campaign), Oracle was most accurate, calling the winner in 80% of the polls in which they were the last to weigh-in on a particular riding.
Telelink, in Newfoundland & Labrador, scored 66% accuracy while CROP clocked in at 58%. Segma scored 50% while Léger Marketing was shutout in the three ridings in which they were the last to report.
Finally, on to independents. During the campaign, I specially dealt with three prominent independents: James Ford in Edmonton - Sherwood Park, Helena Guergis in Simcoe - Grey, and André Arthur in Portneuf - Jacques-Cartier. Hec Clouthier in Pembroke-Nipissing-Renfrew would have been worthy of attention, but there was no way to predict his vote haul without guessing.
For Ford and Arthur, I had applied a reduction in their vote from the previous election by 87.9%. In the few cases where prominent independents ran in subsequent elections, this is the amount of vote reduction I had found.
This performed well. Using the provincial vote totals to get the proportional swing for the other parties, with this factor I would have pegged Ford's vote at 29.1%. He actually got 29.5%.
The model calculated Arthur's vote to be at 27.9% with this factor. He actually got 27.8%.
Guergis was another situation altogether, as she left a party to sit as an independent and then ran as an independent. Looking back on past cases, I had found that the circumstances of an MP's departure played a big role in their ability to retain their vote. Candidates who left for positive principles kept roughly half their vote, those who left for negative reasons kept about a quarter of it. Those who left in disgrace kept about 6% of their vote. Unsure how to classify Guergis, I gave her the average reduction of about 1/3.
With the actual provincial results, the model gave her 22.2% of the vote. She actually got 13.5%. Applying the negative factor, which all-in-all is probably the best way to describe her situation, would have given her about 17% of the vote.
All of these calculations have led to the model being tweaked for the upcoming provincial campaigns. I will use the results of those campaigns as well to do the same post-mortem on the factors for subsequent elections.
Using these federal numbers does work for provincial campaigns. I tested the new Quebec projection model on the 2008 provincial election. The model is the same one used for the 2011 campaign but with these new factors. It had an accuracy level of 94% and an error of 2.5 seats per party.
