Previously, I spoke with Don Mills of Corporate Research Associates (who should be very active in the upcoming Nova Scotia election), Frank Graves of EKOS Research, and Darrell Bricker of Ipsos-Reid.
Léger Marketing has been an active political polling firm in Quebec for a very long time, but has moved over to online in the last few years. This is what Christian Bourque had to say about the methodology:
308: Léger Marketing has used online
polling for some time now. Why was the decision made to move over to that
methodology?
CB:
We made the decision based on the fact that we control the sample. It is
our panel. We control it from recruitment to data collection to data
cleaning. We started the panel in 2004 and felt comfortable using it for
political polling after almost three years of comparative polling telephone to
Web. Really, we have focused on Web-based electoral polling since 2007.
308: What are the advantages of
conducting your polls online instead of over the telephone?
CB: Cost, timing is crucial
and no social desirability bias. We can get up and running faster if the
demand from the client requires quicker turnaround. Rushing telephone
projects can mean “burning” samples to quickly fill quotas. Given that media
clients do not have a lot to spend on polling we have been able to produce
larger samples at a fair cost compared to telephone. The “honesty”
factor also works in favour of the Web.
308: What are the disadvantages?
CB: Controlling for potential
differences between panel members and the general public is always something we
need to control for. Our panel is over 70% RDD recruitment coming
from our call center, so we are already very confident about the source. Panelists here are profiled at length so we can compare them, not
only on socio-demographics but we can do it as well on technology variables,
health-related profiling questions and we could even weight on beer brand to
conform to market share statistics if we wanted to.
308: How is your online panel recruited and what steps
do you take to ensure the sample is representative?
CB: Most of the panel is
recruited from our telephone studies and telephone-based recruitment. It
has a higher cost compared to online recruiting but generates more reliable and
loyal panelists. We profile on over 90 variables over time so get a good
grasp on who these panelists are and we stratify samples at the invitation
stage to ensure that the output will not require important corrections or
weighting. We also have a data cleaning and data quality protocol that
gets rid of speedsters, straightliners and potential fraudsters.
308: What challenges do you face in building a
representative sample of the population, considering that not everyone has
access to the Internet and the potential for opt-in panels to attract a
different sort of respondent?
CB: 86% of Canadians went
online last week. That’s more that households who have a landline
phone. In a market of rapidly decreasing response rates over the
telephone, no methodology should feel they can take the high road and look down
on the others.
308: There are debates in the industry about the
problems surrounding online polling not being probabilistic, despite some good
performances. Why is this, or isn't it, a problem?
CB: Compare the results of
probabilistic vs. non-probalistic polls in BC, Albera, Ontario, Quebec and the last
few federal elections and you will not find a clear pattern of who has done
best or worse. You either feel very strongly about one or other, but it
comes down to faith and preferences more than any clear conclusions one can
reach from historic data.
308: Léger Marketing has a very long history of
polling in Quebec. How has political polling changed over that time?
CB: Like everywhere else,
declining participation in elections is making our work more challenging. How do we account for that 30% to almost 50% who simply do not show up on
election day. Should we move to “likely voter” models in Canada? If
you cross-tabulate participation by age and voting intent by age, you can
explain most of the differences between polls and election results in the
recent past (except Alberta). But age is not the only factor. Disengagement and cynicism need to be factored in too, outside of age.
When doing comparative
polling or comparing our historical results and those of the competition,
differences between phone and web tend to be rather small and not necessarily
consistent over time. We found that only weighting by age and sex will
tend to produce slightly more left-leaning results on the Web. We have
been using a more complex weighting scheme over the past six years to account for
that (education, income and household composition are now factored in).
I'd also like to add the
following:gGiven the recent critique and, some would say, controversy in the
market, I believe we, as an industry, should come to agree to greater
disclosure mechanisms to allow the industry to develop a better understanding of
the changing landscape out there. This would benefit us
all.
**************
Abacus Data is a relative newcomer to the political polling world, having first appeared in 2010. Abacus uses an online panel for its polling, but has used other methodologies in the past. I thought David Coletto could give us an interesting perspective as he has worked in the industry in an era where online polling was always an option, as opposed to the other major players who cut their teeth in the telephone-only age.
308: Why has Abacus settled on
using online panels for political polling, and what are the advantages and
disadvantages of the decision?
DC:
Abacus Data decided to exclusively use online panels for political polling as
we decided that the advantages outweighed the disadvantages.
Advantages
Online
panel research provides for a variety of advantages over live telephone or IVR
research. Online research allows a large
number of respondents to be contacted simultaneously, meaning the study can be
completed much faster than with other methods. Also, the nature of online design allows for great flexibility in the
visual appearance of the survey and in question design. Such design aspects allow for the creation of
scale questions, visual sliders, drag and drop, or even the presentation of
audio and video to respondents. Further,
online research allows for broad flexibility in sample design to target groups
of respondents along virtually any screening criteria. Finally, online research is considerably more
affordable than telephone or IVR, making it attractive for smaller firms and
repeat projects.
Disadvantages
As
online polling involves drawing sample from a panel of potential respondents,
it does not constitute the entire population and therefore cannot be considered
a true random sample – this is the primary disadvantage of online polling. Further, with online research it is currently
not possible to verify that the respondent is exactly who they claim to
be. While this is also true of IVR, it
is much easier to verify an identity with a live telephone interview. Online polling can also result in certain
coverage bias, especially among lower income and older groups.
308: What are the costs of online
polling compared to over the telephone?
DC:
The capital costs of setting up and maintaining a call center are high, just as
building a panel is expensive. However,
licensing costs of online are much more affordable than contracting out to a
call center. Online allows us to be a
full service firm in house and still be small, meaning Abacus is able to
control the research process from beginning to end.
Actually
carrying out the online research requires somewhat less effort than live
telephone, as there is no need for a paid bank of phone interviewers. More importantly, online has far more quality
control, as all responses can be easily monitored as they arrive. Further, there is no need to observe or
control for interviewer bias.
308: What are the challenges faced
in building a representative sample?
DC:
There are challenges, but not around representativeness of demographics or
regional variables, but rather psychographic representation like interest in
politics, political participation and engagement in public issues are likely
greater for anyone who answers a survey. However, large panel management firms have a vested interest in
maintaining quality panels and ensuring that samples are as representative as
possible.
308: There are
debates in the industry about the problems surrounding online polling not being
probabilistic, despite some good performances. Why is this, or isn't it, a
problem?
DC:
It is a problem, but it is something that all survey firms must face. Probability issues become more significant
when respondents are over-surveyed, meaning they change their behaviour or
attitudes because they are surveyed often. We have an in-house policy to screen out frequent survey
participants. Abacus tries to solve the
problem by making the sample as representative as possible, use minimal
weighting, weeding out survey frequent takers, and using high quality large
panels.
The
problem with probability sampling extends to telephone surveys however with
large portions of the population refusing to answer surveys whether because of
increased call screening or because of the refusal to respond.
The
growing use of cell phone or internet based calling will continue to make
telephone surveys more difficult, more expensive, and therefore less
representative.
308: What role does weighting
play in good polling?
DC:
Although weighting plays an important role in helping us to make our samples
representative of the population, our data is not heavily weighted.
We
use balanced sampling and interlocking quotas, similar to stratified sampling
strategy, to ensure that the respondents captured are as representative as
possible and heavy weighting is not required.
Weighting
is particularly challenging for IVR polling, because certain demographics are
more likely to answer the phone. Therefore, IVR surveys bias towards women and older demographics.
308: What are the challenges
involved in building a representative sample of voters, rather than just of the
entire population?
DC:
The number one challenge is trying to predict who will actually vote, as people
are less likely to admit that they do not vote, or that they don’t plan on
voting.
Further,
we know that those who answer surveys are likely to be more engaged than those
who don’t.
To
address these challenges, online research allows us to use varied question
types and measure likelihood to vote in different ways. By being transparent
with the models and try to forecast what the electorate will look like versus
the general population, Abacus attempts to be as clear and accurate as
possible.
In
Canada, this problem is evolving. As a
result of BC and Alberta, we are taking this issue seriously and will test a
number of models in the next election we participate in.
308: As a relatively new polling
firm, what challenges did you face in getting into the market?
DC:
We face a small-c conservative industry that is adverse to change and
innovation, and, quite frankly, are a little threatened by a new crop of
researchers like us that are testing the established ways.
I
think early on we established our credibility by demonstrating that our online
research methodology could accurately forecast the 2011 federal election and
2011 Ontario provincial election.
We,
like many other pollsters, failed to really understand what was going on in
Alberta, using a methodology we no longer use (IVR). In BC, our only poll was conducted before the
leaders debate, so our performance there is difficult to judge.
308: How is the business of
polling evolving?
DC:
I foresee more smaller players emerging.
The
business of polling will be completely online, within the next 10 years nobody
will answer their phone unless they know who is calling, if we are using
telephones at all.
So,
the industry needs to perfect and refine how we conduct internet surveys. As Google is showing us, there will be new
ways to generate sample that are only emerging as alternatives now, many of
which lean towards indirectly observing behavior rather than asking direct
questions.
308: What has to be done to
ensure that online polling can produce good results in the future?
DC:
If you mean being able to predict elections, the question is predicting who is
going to turn out to vote. I do think
online polling is producing good results now for our clients, whether it’s
testing new marketing concepts or public support for policy proposals, or the
potential for new product success.
