Some of the polls have been showing a close race between Liberal candidate Glenn Thibeault and the NDP's Suzanne Shawbonquit, with independent Andrew Olivier earning a big chunk of the vote. Other polls have shown Thibeault with a relatively comfortable lead. What gives?
Forum's first poll on January 13 put the gap at just two points, with Shawbonquit narrowly ahead. Olivier scored just 1%. Then on January 21, the gap was three points for Thibeault as he dropped seven points and Shawbonquit fell 12. Olivier made a miraculous 21-point gain, which Forum called a surge.
Meanwhile, Oraclepoll gave Thibeault a lead of 13 points on January 12-15 at 39% to 26% for Shawbonquit, with Olivier at 19%. On January 20-22, the gap was 16 points (42% to 26%), with Olivier at 20%. In other words, while Forum was showing a major shift, Oraclepoll was showing stability.
So who's right?
There are many reasons to trust Oraclepoll's numbers more than Forum's. Here's why.
Firstly, Forum is based out of Toronto and Oraclepoll is based out of Sudbury. Right out of the gate, the local firm has an advantage both in terms of local knowledge and a local area code showing up on the caller ID.
Secondly, and most importantly, Oraclepoll is conducting a superior survey. Forum is doing its polls via IVR, as it always does. It jumps into the field for a few hours on one evening, gathers its 500 to 800 responses, and reports the numbers a day or two later (response rates being what they are for Forum, it probably needs to call 25,000 to 80,000 households to get even those small samples).
Though it doesn't say explicitly, we can assume that Forum is not calling cell phones. This is because, unlike land lines, it is not possible to know for certain where a respondent on a cell phone is living. This is a problem in by-elections, as ridings do not entirely occupy an area code. This is not an issue in provincial or national polling. Cell phones can be included there without issue, and usually are.
Update: Commenter DL (see below) points out that targeting cell phones in a billing area is straight-forward, so Forum may indeed be calling cell phones. The inability to call cell phones pinpointed to a specific riding would be more of a problem in a city like Toronto where there are multiple ridings within one municipality.
Oraclepoll, on the other hand, is doing its polls with live-callers and it explicitly says that it is including cell phones in its sample (perhaps it has a list of local cell phones to call from). It is also in the field for several days, and according to its press release is calling back numbers where respondents did not answer up to five times before giving up. That is how a poll is supposed to be done.
Forum is getting responses from whoever is home between, say, 7 PM and 9 PM on January 13 and January 21 and is willing to answer a survey. Oraclepoll is doing everything that is reasonably possible to reach everyone it is calling. The potential for a biased sample is far lower.
Lastly, Forum's 1% result for Olivier on January 13 defies logic. At the time the poll was released, people were very surprised that Olivier, who had a relatively high profile, was scoring just 1%. That his number jumped from 1% to 22% in eight days makes little sense, particularly when Oraclepoll was showing his support levels to be steady at between 19% and 20% over the same period. While it is conceivable that the discrepancies in the results for the Liberals and NDP can be explained away by sampling issues and statistical probability, the discrepancy for Olivier's numbers are virtually impossible.
I don't know what happened in Forum's January 13th poll. But it simply doesn't make sense.
In that survey, Forum inquired as to whether Sudburians (Sudburites? Sudburers?) approved of the candidates on offer. The responses there just don't line up with the voting intentions numbers it recorded.
At the time, the poll found that 95% of respondents were aware or had an opinion of Thibeault, compared to 75% for PC candidate Paula Peroni and just 61% for Shawbonquit. But Olivier, who had just 1% support in the same poll, had 88% recognition - better than either the NDP or PC candidates.
That poll also suggested that 48% of all respondents (including undecideds and those who did not know him) approved of Thibeault, indicating that he was converting about 80% of sympathizers into supporters. Shawbonquit, with 41% approval, was converting an incredible 99% of sympathizers.
Peroni's approval was just 28%, as she was unknown to many respondents, but she was converting just 45% of sympathizers into supporters.
Olivier, though, had an approval rating of 60% of all of those sampled - and this is including those who did not have an opinion or did not know him. That was higher than any other candidate. According to the poll, we are thus supposed to believe that Olivier was converting just 2% of his sympathizers into supporters. If you can believe that, I have a bridge that spans the width of Lake Huron to sell you.
I think we can reasonably conclude that the January 13 poll by Forum can be tossed aside and discarded. Forum's January 21 poll has no similar problems, and support levels for Olivier and Peroni are similar to those recorded by Oraclepoll. Considering the sample sizes, Shawbonquit's support is also within the margin of error of these two polls, and Thibeault's is only slightly outside of it.
Due to the advantages that Oraclepoll's survey has over Forum's (not to mention the poor record Forum has in by-elections outside of Toronto), the benefit of the doubt should probably go to Oraclepoll. And that's even with the smaller sample size (the margin of error of decided voters would be just under +/- 6%).
If we use that margin of error to estimate support ranges, we'd get Thibeault at between 36% and 47%, Shawbonquit at between 21% and 31%, Olivier between 16% and 25%, and Peroni between 5% and 11%. That is probably as close as we can get to the truth at this stage of the campaign, which will come to a merciful end on Thursday.
