The gamification of surveys

How can gaming principles be used in research? This is a fascinating area that I know Tom Ewing has been spending some time thinking about.

I haven’t, but a combination of some frustrations on a project and reading this excellent presentation, entitled “Pawned. Gamification and its discontents”, got me thinking specifically about how gaming principles could contribute to data quality in online (or mobile) surveys.

The presentation is embedded below.

The problem

There are varying motivations for respondents to answer surveys, but a common one is economic. The more surveys completed, the more points accrued and money earned.

In its basic sense, this itself is a game. But like a factory production line team paid per item, it promotes speed over quality.

As such, survey data can be poorly considered, with minimal effort going into open-ended questions (deliberative questions are pointless) and the threat of respondents “straight-lining” or, more subtly, randomly selecting answer boxes without reading the questions.

The solution

Some of these issues can be spotted during post-survey quality checks, but I believe simple gaming principles could be used (or at least piloted) to disincentivise people to poorly complete surveys.

Essentially, it involves giving someone a score based on their survey responses. A scoring system will evidently require tweaking to measures and weights over time, but it could consist of such metrics as

• Time taken to complete the survey (against what time it “should” take)
• Time taken on a page before an answer is selected
• Consistency in time taken to answer similar forms of questions
• Length of response in open-ended answers
• Variation in response (or absence of straight lines)
• Absence of contradictions (a couple of factual questions can be repeated)
• Correct answers to “logic” questions

A score can be collected and shared with the respondent at the end of the survey. Over time, this could seek to influence the quality of response via

• Achievement – aiming to improve a quality score over time
• Social effects – where panels have public profiles, average and cumulative quality scores can be publicly displayed
• Economic – bonus panel points/incentives can be received for achievements (such as a high survey quality score, or an accumulation of a certain number of points)

The challenges

For this to work successfully, several challenges would need to be overcome

• Gaming the system – there will always be cheats, and cheats can evolve. Keeping the scoring system opaque would mitigate this to an extent. But even with some people cheating the system, I contend the effects would be smaller with these gaming principles than without
• Shifting focus – a danger is that respondents spend more time trying to give a “quality” answer than giving an “honest” answer. Sometimes, people don’t have very much to say on a subject, or consistently rate a series of attributes in the same manner
• Alienating respondents – would some people be disinclined to participate in surveys due to not understanding the mechanics or feeling unfairly punished or lectured on how best to answer a survey? Possibly, but while panels should strive to represent all types of people, quality is more important than quantity
• Arbitrariness – a scoring system can only infer quality; it cannot actually get into the minds of respondents’ motivations. A person could slowly and deliberately go through a survey while watching TV and not reading the questions. As the total score can never be precise, a broad scoring system (such as A-F grading) should be used rather than something like an IQ score.
• Maintaining interest – this type of game doesn’t motivate people to continually improve. The conceit could quickly tire for respondents. However, the “aim of the game” is to maintain a minimum standard. If applied correctly, this could become the default behaviour for respondents with the gaming incentives seen as a standard reward, particularly on panels without public profiles.

Would it work? I can’t say with any certainty, but I’d like to see it attempted.

sk

Advertisement

Links – 21st November 2008

My top 10 reads of the past week:

1. The Times published an absolutely fantastic article looking at neuroscience and how we can improve our brain performance. The writer pays short shrift to the DS Brain Training activities, for the sensible reason that this rewards recognition and repetition over learning. While we do not yet know a lot about our brain, the author exhorts us to work on improving oneself through a simple mantra: Pay Attention

2. On a neuroscience theme, Martin Lindstrom – author of Buyology – has an article on Advertising Age explaining why sponsorship of American Idol works for Coke but not Ford. Essentially, Ford has had trouble justifying its existence.

3. How intelligence can overcomplicate: Students trying to predict the stockmarket perform worse than a rat finding a piece of cheese. It is the conflict between striving for perfection (through modelling) or accepting a reasonable chance of success (Science Blogs)

4. Chris Anderson has conceded that the Long Tail argument is flawed, in that the number of aggregators providing the long tail of product options conform to powerlaws (think Google, Amazon or Netflix)

5. ETH Zurich have studied Youtube videos to try and work out what constitutes a successful upload. Their typology consists of viral, quality and junk videos – a more nuanced approach to my 4-video typology where viral constituted a single element (against reference, scheduled and topical) (Newteevee)

6. Engage Research and Global Market Insite have published a report saying that online surveys bore respondents. Quite. Unlike telephone or face to face interviews, online is restricted to the narrower range of those that opt-in. Therefore things need to be mixed up regularly in order to avoid a) burn-out and b) recognition of formats and patterns. (Brand Republic)

7. Fast Company has a profile of Sam Ewan – whom some people may refer to as a guerrilla marketing. I don’t particularly like the label, but I think the concept is fantastic – the levels of creativity in constructing a unique experience are limitless

8. A NY Times article looks at how industries change to survive e.g. one might predict the extinction of the bicycle with the advent of the automobile but that evidently wasn’t the case

9. Lifehacker tells us how to burn any type of video file to a playable video DVD

10. And finally, a triumvirate of brilliant little websites (OK I’m cheating in order to get a nice round figure of 10). Tag galaxy transposes Flickr searches to a galaxy of interrelated search items, the Charlian is a Charlie Brooker themed Guardian that came out of their hack day, and Let me Google that for you gives a visual display of searching to colleagues lazily shouting out a question when the answer is in front of them

sk