How representative are Online surveys?

To answer the above question in three words: I don’t know.

Generally, I have been sceptical about the relative veracity of Online surveys. Working for a media owner, there is a general concern that moving surveys online may reduce the strength of TV and increase that of the Internet. But I am being won around.

After all, no methodology is perfect. In fact, it could be argued that all are inadequate. Even if one were able to take a census of the entire population (even the UK census only has a 94% response rate), how accurately are people able to express their unconscious thoughts, desires and opinions?

Which is why I was pleasantly surprised to read that YouGov correctly predicted the results of the London Mayoral election. Accurate polling always requires a bit of luck. When I worked at a research agency, the weighting factors for the forthcoming election were changed at the last moment, and fortunately improved the prediction. But even when considering the fluctuations, it does represent a significant victory for the online method.

There are far better resources than this blog debating the relative merits and drawbacks of research methodologies, but my softening of opinion has come about for two main reasons that I have recently given more thought

Societal changes are making the traditional methodologies less accurate over time: The rise of the one person household makes it more difficult for face-to-face interviewers to catch people at home at a time where they are willing to participate. Telephone research is becoming less representative thanks to the rise of mobile phones at the expense of landlines, and the popularity of the TPS. Even if mobile numbers are included in the sample, people are far less willing to participate, since mobile phones are more personal and the call is therefore more intrusive. And while the TPS doesn’t cover market research, some companies voluntarily clean the sample of TPS numbers, since the public perception is that research is no different to telemarketing. And as online penetration increases, one would expect survey representativeness to follow suit.

Online research is more conducive to considered opinion: Online surveys produce more honest responses thanks to the anonymity provided. Without an interviewer waiting for an answer, the respondent can also give a more considered answer (if they so desire). Combined, these will produce more accurate data.

Of course, these points aren’t uniformly positive. Even though Internet access increases, the proportion of those actively on a research panel will still be quite small. Gritz (2004) achieved an 8.4% sign-up rate for an online survey and I wouldn’t be surprised if this figure would be lower if the experiment were repeated now. And Online surveys may allow for more considered responses but without an interviewing probing, the answers may be ambiguous and thus meaningless. But, for me at least, the benefits far outweigh the drawbacks.

I do still have one major concern with online surveys. Without any proof, I have the perception that the attitudinal differences between those that take part in online surveys and those that don’t is greater than the differences between those that do and don’t respond in different methodologies. Those that join online panels are self-selecting, and will tend to spend more time online than the average person.

Sticking with YouGov, their Brand Index (which, in general, I like) ranks Google and Amazon as the top 2 brands in 2007. Would they still come out on top in an offline survey, factoring in the third that don’t use the Internet and those that spend more time with traditional media? I’m not so sure.

But for me at least, I have far less reservations with moving research online than I had a year ago.

I’d be interested in hearing other people’s thoughts on the benefits and drawbacks of the shift. Am I late to the party in accepting online, or do others still hold reservations?

sk

Advertisements

6 Responses

  1. Hi Simon

    Haven’t got lots of time to comment I’m afraid. However, there are still some sceptics of online as a method, but many forward thinking researchers are realising that as internet penetration has risen then representativeness has improved. Of course, if you are looking to survey just OAP’s then online might not be the best route! As you point out, other ‘trad’ techniques have their own drawbacks. For me, the key is to tailor the contact method dependent on the audience and the nature of the research. (As an aside, online can be used to cost effectively interview, but an omnibus survey can help provide a rep profile with which to set quotas/weights.)

    This might raise a smile; my colleague Ray ran a poll on Facebook re: London mayor election. He was more accurate than Ipsos Mori (who were terribly (worryingly?) wrong).

    http://www.virtualsurveysdiscussion.com/vslblog/?p=87

    Cheers
    Jon

    PS keep up the blog!

  2. Hi Jon

    Thanks for commenting, and for the encouragement. It is remarkable that Ray was closer to Ipsos MORI in predicting the results. As well as refuting the notion of Internet users being more liberal than average, it just goes to show that with all the sampling and weighting experience in the world, polling is as much a (dark) art as a science.

    I think you are dead right in that the audience and subject need to be at the forefront when considering a methodology. For media and tech, there is still that worry that online surveys overemphasise the interest in new media

    All the best
    Simon

  3. Jon and Simon:

    Thanks for pointing out Ray’s Facebook study. At BrainJuicer we too believe that the “crowd is wise”, which is the basis of our Predicitive Markets methodologies and other ways we look at squeezing accuracy and reliability from online sample. Our own little study to probe for the winner in the London mayoral race, was accurate (sorry Ipsos Mori) and also revealed the “whys” and the repsondents’ assessment of the emotional connection to the decision as well.

    http://www.brainjuicer.com/download/MayorofLondonJuicer_May2008.pdf

    Debates about on-line will rage on, but research will continue to require innovation beyond simply online vs. offline, as issues of sampling become more complex. Where will that innovation, and the next paradigm of reliability, come from? We would submit that it will come from those research agencies willing to experiment. It will not emerge from agencies merely clinging to older methoidologies that are/may be less relevant as technology adoption changes the landscape of consumer insight.

    Thanks for stimulating the dialog,

    Susan

  4. Hi Susan, thanks for commenting.

    Experimentations in not only methodologies but also calibrations will definitely be how the industry progresses. The Internet has revolutionised not only the industry but our lives, and we need to move on to reflect that.

    But accurate representation remains the holy grail. Progress is being made, but I see us as still being a long way from being able to account for all types of people within online research

    I have a lot of respect for BrainJuicer, and hope you continually experiment with, and improve, the medium.
    Simon

  5. I was a member of the online survey site called ask500people.com. I am a conservative. I quickly learned that the moderators of the website were liberal. I observed that a great deal of the comments that were left on the site that reflected a conservative point of view were “moderated down.” In other words, the moderators of the site masked the comments so that they could only be read by clicking on a hotlink. I’m not talking about offensive comments, either! Many, if not most, of the comments were clear, concise, and respectful in nature.

    On the other hand, I’ve observed comments that leaned strongly to the liberal side of things, even when they contained offensive or inappropriate language and subject matter, were not “moderated down.”

    Additionally, I’ve noticed that many of the people on the site, myself included, who posted conservative views, eventually got banned (or mysteriously stopped posting).

    So, I’d have to say that when considering results of online polls and the like, one has to do so cautiously – considering who runs the site and what their level of participation or “moderation” might be with the members and their views.

  6. Hi Raymond, thanks for stopping by.

    Your story saddens me. Subconscious moderator/interviewer bias is something that we are all aware of, but your experiences sound like they are being more overt. If communities aren’t allowing balanced and fair discussion, then you are absolutely right in questioning their level of inclusiveness.

    Best
    Simon

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: