How representative are Online surveys?

To answer the above question in three words: I don’t know.

Generally, I have been sceptical about the relative veracity of Online surveys. Working for a media owner, there is a general concern that moving surveys online may reduce the strength of TV and increase that of the Internet. But I am being won around.

After all, no methodology is perfect. In fact, it could be argued that all are inadequate. Even if one were able to take a census of the entire population (even the UK census only has a 94% response rate), how accurately are people able to express their unconscious thoughts, desires and opinions?

Which is why I was pleasantly surprised to read that YouGov correctly predicted the results of the London Mayoral election. Accurate polling always requires a bit of luck. When I worked at a research agency, the weighting factors for the forthcoming election were changed at the last moment, and fortunately improved the prediction. But even when considering the fluctuations, it does represent a significant victory for the online method.

There are far better resources than this blog debating the relative merits and drawbacks of research methodologies, but my softening of opinion has come about for two main reasons that I have recently given more thought

Societal changes are making the traditional methodologies less accurate over time: The rise of the one person household makes it more difficult for face-to-face interviewers to catch people at home at a time where they are willing to participate. Telephone research is becoming less representative thanks to the rise of mobile phones at the expense of landlines, and the popularity of the TPS. Even if mobile numbers are included in the sample, people are far less willing to participate, since mobile phones are more personal and the call is therefore more intrusive. And while the TPS doesn’t cover market research, some companies voluntarily clean the sample of TPS numbers, since the public perception is that research is no different to telemarketing. And as online penetration increases, one would expect survey representativeness to follow suit.

Online research is more conducive to considered opinion: Online surveys produce more honest responses thanks to the anonymity provided. Without an interviewer waiting for an answer, the respondent can also give a more considered answer (if they so desire). Combined, these will produce more accurate data.

Of course, these points aren’t uniformly positive. Even though Internet access increases, the proportion of those actively on a research panel will still be quite small. Gritz (2004) achieved an 8.4% sign-up rate for an online survey and I wouldn’t be surprised if this figure would be lower if the experiment were repeated now. And Online surveys may allow for more considered responses but without an interviewing probing, the answers may be ambiguous and thus meaningless. But, for me at least, the benefits far outweigh the drawbacks.

I do still have one major concern with online surveys. Without any proof, I have the perception that the attitudinal differences between those that take part in online surveys and those that don’t is greater than the differences between those that do and don’t respond in different methodologies. Those that join online panels are self-selecting, and will tend to spend more time online than the average person.

Sticking with YouGov, their Brand Index (which, in general, I like) ranks Google and Amazon as the top 2 brands in 2007. Would they still come out on top in an offline survey, factoring in the third that don’t use the Internet and those that spend more time with traditional media? I’m not so sure.

But for me at least, I have far less reservations with moving research online than I had a year ago.

I’d be interested in hearing other people’s thoughts on the benefits and drawbacks of the shift. Am I late to the party in accepting online, or do others still hold reservations?

sk

Advertisement