Forum tried to poll PC party members. What happened next was strange.

Polling for a party's leadership campaign is fraught with hazard. While it can reveal how the general population and party supporters feel about the leadership candidates - and that is useful information - they can't tell us anything specific about the actual outcome. Only a very small proportion of party supporters actually vote in a leadership race, and that proportion is even smaller when we are talking about the general population.

There are two ways to go about polling a party's membership. One way is to obtain a list of party members and call them, which has happened in the past. The less common route is to do a standard poll and ask people if they are party members.

That's what Forum did in its latest poll concerning the Ontario Progressive Conservative leadership race. Good for them for trying, but logistically it should be a nightmare. According to the PCs, they have over 80,000 members, representing 0.6% of Ontario's adult population. You'd have to do a poll of 50,000 people to get a decent sample size of 300 party members, if the poll is done properly. With the response rates that IVR gets, you'd have to call about 2.5 million households.

Instead, Forum polled 881 Ontarians and found 65 paid-up party members. This is revealing, for several reasons.

Those 65 paid-up party members represent 7.4% of the sample, quite a bit more than what it should have been if the poll was completely random. Forum found 12 times as many PC party members in its poll than it should have.

This raises a few issues. First is the question of whether we can believe these 65 respondents. If the poll was done correctly, some of them would have been lying or misunderstood the question.

Second is the question of accuracy. If Forum could find 12 times as many PC members as it should have, what does that say about its sampling methods?

The third question is, perhaps, the most interesting one. What if active political partisans are just so much more likely to participate in a poll than the rest of the population, that being over-represented by a factor of 12 is actually not a sign of a problem with the poll, but a problem with the people who take part in a poll in the first place?

Response rates are tiny these days, and particularly with IVR polling. Nova Scotia's electoral laws require pollsters to report their response rates, and when Forum did in 2013 it showed that rate to be about 2% to 3%. That is a large proportion of potential voters not bothering to pick up the phone or complete the survey.

If that 2% to 3% is largely made up of people who are active partisans, does that skew the poll? Other polling has shown that poll-takers are more likely to be politically active and do volunteer work. The important question, then, is whether these people are politically different than the rest of the population. If they are, than the poll will be led astray. If they aren't, than the poll will be effective.

Putting aside these existential questions, the poll itself had a counter-intuitive result. PC party members broadly lined-up with the views of the general population and of PC supporters.

Of those 65 people who claimed to be party members, 51% said they preferred Christine Elliott. Another 13% said they preferred Monte McNaughton, while 10% opted for Patrick Brown. In the chart above, each box represents one of those 65 respondents.

Another 14% said none of these three contestants were their preference, while 11% were undecided.

Among PC supporters, a larger group, Elliott had 35% to 15% for McNaughton and 11% for Brown. Among all Ontarians, Elliott had 24% to 7% for McNaughton and 6% for Brown.

There are many caveats with these sorts of numbers, some of which were spelled out above. But the most important is, of course, sample size because the margin of error of a sample of 65 people is +/- 12.2%.

When we look at the poll with these margins of error, we get a much less clear picture.

(This is also a good example of how the margin of error applies differently depending on each candidate's support. Brown, at 10%, cannot be as low as -2%.)

Now we see Elliott at between 39% and 63%, McNaughton between 5% and 21%, and Brown between 3% and 17%.

If we look at just the decided PC party members, the margin of error increases to +/- 14.1%, and Elliott's support ranges between 55% and 83%, compared to 7% to 29% for McNaughton and 4% to 24% for Brown.

By these numbers, we would consider Elliott a lock to win on the first ballot. McNaughton and Brown are fighting for second place. However, we also have a pretty good idea that this is not the actual contours of the race.

We won't know the final tally of each candidate's membership drive until the end of April, when the PCs release the numbers. But Brown has claimed about 40,000 members, with Elliott claiming 34,000 and McNaughton 20,000. One oft-quoted source puts McNaughton closer to 6,000, which is a more intuitive result. We just don't know for sure, though any misleading candidate will be embarrassed once the numbers are announced.

So the numbers are probably not too far off, particularly if McNaughton is at 6,000 instead of 20,000 (the three claims add up to 94,000, but 80,000 if the source is correct).

At those rates, Brown should be somewhere between 43% and 50% of newly signed-up members, compared to 36% to 43% for Elliott and 8% to 21% for McNaughton.

This ignores the voters that were already with the party when the race started (some 10,000) and those signed up by Vic Fedeli and Lisa MacLeod (both of whom endorsed Elliott).

So now we get to the question of accuracy. Considering the limitations of Forum's poll, their Elliott numbers are, perhaps, plausible. Those undecideds could be from the ranks of the original party members, who should represent some 13% of the total. But to have Brown and McNaughton within the same error bands with at most half of Elliott's support, and McNaughton narrowly favoured among the two, does not quite align with what we know about the race.

This really puts the poll in question, and highlights the limitations of this attempt at getting at the voting intentions of party members. Does Elliott really have the kind of advantage that Forum gave her? It doesn't seem so, but we won't know for sure until the votes are counted in May.