The gamification of surveys

How can gaming principles be used in research? This is a fascinating area that I know Tom Ewing has been spending some time thinking about.

I haven’t, but a combination of some frustrations on a project and reading this excellent presentation, entitled “Pawned. Gamification and its discontents”, got me thinking specifically about how gaming principles could contribute to data quality in online (or mobile) surveys.

The presentation is embedded below.

The problem

There are varying motivations for respondents to answer surveys, but a common one is economic. The more surveys completed, the more points accrued and money earned.

In its basic sense, this itself is a game. But like a factory production line team paid per item, it promotes speed over quality.

As such, survey data can be poorly considered, with minimal effort going into open-ended questions (deliberative questions are pointless) and the threat of respondents “straight-lining” or, more subtly, randomly selecting answer boxes without reading the questions.

The solution

Some of these issues can be spotted during post-survey quality checks, but I believe simple gaming principles could be used (or at least piloted) to disincentivise people to poorly complete surveys.

Essentially, it involves giving someone a score based on their survey responses. A scoring system will evidently require tweaking to measures and weights over time, but it could consist of such metrics as

• Time taken to complete the survey (against what time it “should” take)
• Time taken on a page before an answer is selected
• Consistency in time taken to answer similar forms of questions
• Length of response in open-ended answers
• Variation in response (or absence of straight lines)
• Absence of contradictions (a couple of factual questions can be repeated)
• Correct answers to “logic” questions

A score can be collected and shared with the respondent at the end of the survey. Over time, this could seek to influence the quality of response via

• Achievement – aiming to improve a quality score over time
• Social effects – where panels have public profiles, average and cumulative quality scores can be publicly displayed
• Economic – bonus panel points/incentives can be received for achievements (such as a high survey quality score, or an accumulation of a certain number of points)

The challenges

For this to work successfully, several challenges would need to be overcome

• Gaming the system – there will always be cheats, and cheats can evolve. Keeping the scoring system opaque would mitigate this to an extent. But even with some people cheating the system, I contend the effects would be smaller with these gaming principles than without
• Shifting focus – a danger is that respondents spend more time trying to give a “quality” answer than giving an “honest” answer. Sometimes, people don’t have very much to say on a subject, or consistently rate a series of attributes in the same manner
• Alienating respondents – would some people be disinclined to participate in surveys due to not understanding the mechanics or feeling unfairly punished or lectured on how best to answer a survey? Possibly, but while panels should strive to represent all types of people, quality is more important than quantity
• Arbitrariness – a scoring system can only infer quality; it cannot actually get into the minds of respondents’ motivations. A person could slowly and deliberately go through a survey while watching TV and not reading the questions. As the total score can never be precise, a broad scoring system (such as A-F grading) should be used rather than something like an IQ score.
• Maintaining interest – this type of game doesn’t motivate people to continually improve. The conceit could quickly tire for respondents. However, the “aim of the game” is to maintain a minimum standard. If applied correctly, this could become the default behaviour for respondents with the gaming incentives seen as a standard reward, particularly on panels without public profiles.

Would it work? I can’t say with any certainty, but I’d like to see it attempted.

sk

Advertisement

Complacency and Incentives

We know complacency is a bad thing, but we are all guilty of it. But while we may not always look both ways before crossing a road, hopefully few of us are complacent to extent that major repercussions are created. But the nature of complacency means we aren’t always aware when we are doing it.

Incentives are an important part of business. These don’t have to be financial, but they have to be aligned to the long-term objectives of the organisation. When they aren’t, trouble can be caused.

A couple of things have left me pondering over complacency and incentives recently. The first is the rather obvious example of the financial crisis. Despite studying economics as part of my degree, the situation is far too big and complex for me to even begin to grasp – I’ve only recently got my head around what an ISA is.

But when you see terms like “too big to fail” and “golden parachutes” bandied around, you know that something is wrong. Short-termism appears to have been prioritised over long-term growth due to the bonus schemes of the interested parties, and people such as Umair Haque are arguing that capitalism as we know it should be destroyed. But, again, this is not something I fully understand.

I know a bit more about football (association, though my gridiron knowledge is slowly improving), and the team I support has just been relegated. And they fully deserved it.

Ever since Bobby Robson was fired for finishing 5th, the team has been a joke. Overpaid and over-the-hill players were brought in to accompany Newcastle’s innate ability to recruit inept players from foreign leagues. With their huge contracts (15 players are reputedly on more than £50,000 per week – and these contracts are usually after tax) dwarfing any win bonus, and with seemingly little pride on show, there was little incentive for the players to perform (see also: Winston Bogarde). Particularly since they are guaranteed to earn the same amount of money even after relegation.

The players were complacent; they believed their own hype and viewed their wages as symbols of their skills. When the rot set in, nothing could be done to motivate the players to increase their efforts.

I (hopefully) don’t have been problem. As has been noted, this blog has been quieter recently. Aside from weekends away, this is because I’m working longer hours, and my job is a bigger priority than this blog. This isn’t a complaint – I’m having a great time. But in a company of 12 people that largely undertakes ad hoc commissions, there is no safety net and nowhere to hide. If we don’t perform, we don’t earn.

I’m not complacent because my incentive is to contribute to keeping the company in business, and thus keep a job.

How many of the companies undergoing traumatic experiences at the moment have been complacent in the past? As companies look to restructure, they should also look to why people want to work for them. Is it the money, or the desire to succeed? Is it aligned with the company objectives, or isn’t it?

sk

Image credit: http://www.flickr.com/photos/29487767@N02/

http://www.flickr.com/photos/29487767@N02/

Links – 31st October 2008

Rather than a lop-sided list, here is a numbered (but unordered) overview of blogs, articles and tools I have enjoyed over the past few days:

1. Iqbal Mohammed asks a very interesting question: Could price become a skeuomorph? This segues in with an interesting point Matt Rhodes made regarding incentives – it changes the transaction from market to social. Would people prefer to pay in order to avoid any potential obligations in future – whether signing up to a mailing list or being forced into a timeshare? It also reminds me of the Severn Bridge, with its one way tolling. The Welsh like to point out that they can cross over into England for free, but are forced to pay to leave and go back home.

2. There has been an interesting teté a teté on network effects between Nicholas Carr and Tim O’Reilly. While the argument is largely focused upon semantics, several interesting issues are raised. And not entirely unrelated, Faris links to a video of Clay Shirkey talking about Solidarity Goods – items that increase in value as more people use them.

3. Japan’s census looks great. Rather than a straightforward survey, everyone also has to fill in a time diary showing what they are doing throughout the day. Tokyo Tuesday is a fascinating look at the aggregated lives of the salarymen, cyberpunks, samurai and other assorted Japanese stereotypes that one might want to throw into the mix.

4. Can Hulu’s (No.4 on Time’s inventions of the year) current single-ad-per-break model continue? If they are buying content in and paying for their bandwidth then I’m guessing not. Premium positioning only stretches so far. But product placements and other new formats can mitigate the losses from restricting inventory (New York Times)

5. A study commissioned by the Association of National Advertisers suggests that the brand does not influence business decisions in two thirds of the companies questioned. CMOs with an axe to grind against the beancounters, or a realistic assessment of current corporate thinking? (Marketing Charts)

6. A bit old now but Brian Solis has a very thorough overview of the state of Social Media in 2008, and the outlook for 2009

7. An argument in favour of training over talent (CNN) – sounds good to me. An ability to remember trivia can only take one so far in life

8. A digital planning checklist (Katie Chatfield). Does exactly what it says on the tin. Succinct and insightful.

9. Mashable has a nice collection of social media gurus owning up to their biggest mistakes

10. Seth Godin has put the presentation of his latest book – Tribes – onto Slideshare

11. An epic attempt at integrating all of one’s personal data (Kiwi Tobes)

12. A tool that lets you look at the traffic statistics for Wikipedia pages

Enjoy…

sk