The gamification of surveys

How can gaming principles be used in research? This is a fascinating area that I know Tom Ewing has been spending some time thinking about.

I haven’t, but a combination of some frustrations on a project and reading this excellent presentation, entitled “Pawned. Gamification and its discontents”, got me thinking specifically about how gaming principles could contribute to data quality in online (or mobile) surveys.

The presentation is embedded below.

The problem

There are varying motivations for respondents to answer surveys, but a common one is economic. The more surveys completed, the more points accrued and money earned.

In its basic sense, this itself is a game. But like a factory production line team paid per item, it promotes speed over quality.

As such, survey data can be poorly considered, with minimal effort going into open-ended questions (deliberative questions are pointless) and the threat of respondents “straight-lining” or, more subtly, randomly selecting answer boxes without reading the questions.

The solution

Some of these issues can be spotted during post-survey quality checks, but I believe simple gaming principles could be used (or at least piloted) to disincentivise people to poorly complete surveys.

Essentially, it involves giving someone a score based on their survey responses. A scoring system will evidently require tweaking to measures and weights over time, but it could consist of such metrics as

• Time taken to complete the survey (against what time it “should” take)
• Time taken on a page before an answer is selected
• Consistency in time taken to answer similar forms of questions
• Length of response in open-ended answers
• Variation in response (or absence of straight lines)
• Absence of contradictions (a couple of factual questions can be repeated)
• Correct answers to “logic” questions

A score can be collected and shared with the respondent at the end of the survey. Over time, this could seek to influence the quality of response via

• Achievement – aiming to improve a quality score over time
• Social effects – where panels have public profiles, average and cumulative quality scores can be publicly displayed
• Economic – bonus panel points/incentives can be received for achievements (such as a high survey quality score, or an accumulation of a certain number of points)

The challenges

For this to work successfully, several challenges would need to be overcome

• Gaming the system – there will always be cheats, and cheats can evolve. Keeping the scoring system opaque would mitigate this to an extent. But even with some people cheating the system, I contend the effects would be smaller with these gaming principles than without
• Shifting focus – a danger is that respondents spend more time trying to give a “quality” answer than giving an “honest” answer. Sometimes, people don’t have very much to say on a subject, or consistently rate a series of attributes in the same manner
• Alienating respondents – would some people be disinclined to participate in surveys due to not understanding the mechanics or feeling unfairly punished or lectured on how best to answer a survey? Possibly, but while panels should strive to represent all types of people, quality is more important than quantity
• Arbitrariness – a scoring system can only infer quality; it cannot actually get into the minds of respondents’ motivations. A person could slowly and deliberately go through a survey while watching TV and not reading the questions. As the total score can never be precise, a broad scoring system (such as A-F grading) should be used rather than something like an IQ score.
• Maintaining interest – this type of game doesn’t motivate people to continually improve. The conceit could quickly tire for respondents. However, the “aim of the game” is to maintain a minimum standard. If applied correctly, this could become the default behaviour for respondents with the gaming incentives seen as a standard reward, particularly on panels without public profiles.

Would it work? I can’t say with any certainty, but I’d like to see it attempted.

sk

Advertisement

My name is my name

So says Marlo Stanfield. And he has a point.

Reputation means a lot. But reputation is about perception, and there are multiple perspectives in which it can be viewed.

Broadly, reputation can be thought of in four inter-related spheres

• Yourself – your personal brand
• Your organisation (this itself can have several facets, if your organisation is part of a larger conglomerate or affiliation)
• Your industry
• The wider public

Marlo is concerned with his personal reputation among people in the industry – “the game”. He isn’t so worried about the other facets.

With the prominence of polling in the upcoming general election, the research industry is contemplating its reputation among the wider public.

I don’t think it really matters.

This election is more partisan and contentious than any I recall (most likely driven by the likelihood of change, rising prominence of online media giving a voice to more people, and the novelty of the leadership debates). Pot-shots, such as those against YouGov, are inevitable. This article from Research Live shows how YouGov aren’t doing themselves any favours in their need for speed (and this is leaving aside their associations with The Sun/Murdoch/Conservative Party).

I don’t think it matters because the research industry is rarely public facing – the only publicity it really receives is through political polls and PR research.

I’ve written about the problems with PR research in the past, but there is evidently a market for it and so the method prospers. It might damage the reputation of the industry to the wider public but outside of recruitment  (of staff and respondents/participants) it isn’t really relevant.

As Marlo noted, it is industry reputation – for yourself and your organisation – that really matters.

It is similar to the advertising industry. Successful companies have a lot of brand equity through the quality and associations of their work – Wieden & Kennedy and Nike, Fallon and Cadbury, HHCL and Tango, and Crispin Porter & Bogusky and Burger King, to give but four examples.

But what proportion of the general public has heard of these companies, let alone recognises and appreciates their work? Not many. Is it a damning indictment of the strength of the marketing industry that it fails in promoting the most basic thing – itself? Not really. Companies attract talent and business through their successes and image – public perception doesn’t factor.

Ray Poynter is rightly concerned with the the ethics of market research but for me, the importance of this is in maintaining business links. There is no adequate means of policing the research industry – anyone can knock on a door and say they are doing a survey – so it is not a battle worth fighting.

Companies stand and fall by the quality of their work – or at least the perception of it within the industry. Sub-standard work that is openly criticised will only harm long-term prosperity.

Self-regulation and recognition, whether through a recognised body like the Market Research Society, or at a more ad hoc level, can achieve this through highlighting good and bad practice.The research industry needs to be more vocal in showcasing good work, and castigating poor work.

This in turn will filter to the individual level, where the talented and ambitious will compete to work for the top companies. This in turn strengthens the work, and thus the industry. It could even permeate to the public.

There is no quick fix to improve the standing of an industry, and in some cases it isn’t necessarily desirable. Rather than look to the big picture, we should focus on the more immediate challenges.

If we all concentrate on undertaking the best possible work, then a strong reputation – for ourselves, our organisation and our industry – will follow.

sk

NB: The clip of the scene with the quote is below (it is from Series 5, so beware of potential spoilers)