Ten things I learned from the New MR Virtual Festival

My previous post included all of the notes I took while listening into the New MR Virtual Festival. This post contains the key things I took away from the day, and have subsequently been mulling over in the 2 months (!) since.

NB: All slides shown below are taken entirely without permission. If a creator objects to its use, please contact me and I’ll remove it.

1. The boundaries between participation and observation can (and, in some circumstances, should) be blurred

Although Ray Poynter touched on the range of methods in which research can overtly or covertly be conducted online, Netnography (cf. Robert Kozinets) is – to me – one of the more interesting. To be effective, it needs to have the research both participate and observe the environment of interest.

Erica Ruyle argues that observation (or lurking) is fine in the initial stage, since the norms and cultures need to be understood and respected. But active participation is vital in order to get than “insider” knowledge and to be able to read between the lines of the interactions.

This is a difficult proposition to promote commercially as a) the time investment (and thus cost) required will be huge and b) the researcher will need to have a personal as well as professional interest in the topic in order to both be accepted by the community and accept the community. For instance, how many researchers would turn their nose up at being asked to take part in World of Warcraft for 6 months?

Nevertheless, in the right circumstances it could prove to be a fascinating and rewarding exercise.

2. Convenience samples can still be worthwhile

It is true that all non-census research incorporates a convenience sample to some extent. But some methods require more convenience (and thus are less representative) than others.

Annelies Verhaeghe highlighted some of the issues to be aware of when conducting social media research – particularly that we should resolve ourselves to not always know who we are speaking to or monitoring.

Furthermore, something I had not considered but makes sense is that even though companies trumpet the volume of data they scrape and collect, only a sub-sample of that will be analysed due to the diminishing returns of going deeper into a very large data set.

If we’re able to augment social media research with other techniques or data sources – Annie Pettit mentioned some of the benefits of combining social media research with surveys – then it can be a very valuable and insightful method of getting real-world information on past actions and thoughts.

3. Respondents shouldn’t necessarily be viewed equally

Both Rijn Vogelaar and Mark Earls talked about premises realised more thoroughly in their books – The SuperPromoter and Herd respectively.

Segmenting the audience isn’t a new phenomenon – we often restrict our universes to who we are interested in – but within these universes perhaps we should pay more attention to some individuals more than others – particularly given the complex social interactions that cause ideas and opinions to spread. I’m not clever enough to be able to incorporate full network theories into any of my research – in the manner of Duncan Watts, for instance – but perhaps there is an argument for applying simple weights to some projects, to account for some opinions becoming more important than others. Or perhaps it is too contentious to implement without proper academic grounding and proof.

4. More of an effort needs to be made to meet respondents on their terms

Betty Adamou joined the likes of Stowe Boyd in saying that email is on the decline among younger people. This is problematic for online surveys, given that online panels are predominantly administered via email. Given the trends, perhaps we should be looking to Facebook, Twitter, instant messenger etc for both initial recruitment of these audiences and then allow them to dictate how we can contact them to alert them with new surveys. I’m not sure whether a note on a Facebook group could be as effective as an email, but it is certainly worth exploring.

5. Survey structures can do more to take advantage of the online environment

Our media and communications channels have fragmented but the content providers retain a centralised hub of controlling activity. Why can’t the research industry do this? John Dick talked through Civic Science’s modular research methodology, whereby questions are asked in chunks of two or three at a time, but combined at the back-end to build up extensive profiles of respondents.

This approach makes intuitive sense. In face to face research, the main cost was in finding people to speak to. Thus, once they were located, it was efficient to collect as much information as possible. The web is abundant with people, who are time-poor. The cost isn’t in finding them, it is keeping them. People could easily answer three questions a day if there was the possibility of a prize draw. They would be less willing to spend 30 minutes going through laborious and repetitive questions.

There are clearly downsides to this method and plenty of issues to overcome regarding data quality assurances, but the notion of Facebook users answering a couple of questions a day sounds like a feasible way to collect information among people who might be unwilling to sign up to an online survey

6. Surveys don’t have to be boring or static…

Another aspect of the online world that should be explored further is the level of interactivity. Many online surveys are straight ports of face to face surveys – a shame when there are so many more things that a web survey can – in theory – be capable of.

Jon Puleston of GMI highlighted several of their experiments in this area. Interestingly, although interactive surveys take longer, respondents are more engaged, enjoy them more and give “better” answers. I particularly like the idea of timing respondents to give as many answers as possible within a given timeframe. This appeals to people’s competitive nature, and means they’d spend far longer on it than they normally would.

The King of Shaves case study was very interesting. Rather than a 1 minute introduction for a 10 minute survey, this example reversed the process. People were given a detailed briefing on the role of a copywriter, and asked to come up with a creative slogan. In subsequent testing, seven “user-generated” ideas scored better than the advertising agency.

7. But we should be aware of the implications of survey design on data capture…

Jon’s examples showed how framing questions can improve data collection. But Bernie Malinoff warned us that even minor superficial changes to a survey can have a big impact on how people answer questions. For instance, the placement of the marker on a slider scale can heavily impact the distribution of answers.

Bernie also had some important lessons in survey usability – ranging from the wordiness of the questions (something I’ve been guilty of in the past) to the placement of error messages and how they can influence subsequent responses.

Surprisingly, his research found that survey enjoyment was comparable among people who did traditional “radio button” style surveys versus richer experiences, and that people were less willing to take part in future after having completed a flash-based survey.

It acts as a sobering counter-point to Jon’s presentation, but I inferred some caveats to this research (or perhaps I am only seeing what I want to see). I suspect some of the resistance to flash might be down to the “newness” of the survey design rather than a genuine preference for radio-button style surveys. Similarly, design iterations aren’t neutral – I wouldn’t mind different results so long as I felt they were “better” (and any methods to discourage survey cheaters are welcome). Nevertheless, it an important reminder that a better designed survey is only an improvement if it makes the survey more usable and easier to understand, and I completely agree with the final point that the industry should reinforce best practices for interface design.

8. …And whether they are suitable for the audience

Tom Ewing’s talk on gaming and research covered many interesting points, but the one that stuck with me is that it isn’t for everyone. As he points out, FourSquare holds little appeal to him (unless he wanted to be Mayor of his child’s nursery). Similarly, while the number of gamers is rising, it isn’t everyone and so we cannot assume that introducing interactive, exploratory or personalised experiences will automatically make respondents enjoy our surveys more.

Particularly since games design is pretty hard – Angry Birds and Farmville may look simple, but I wouldn’t expect any research agency to devise and incorporate something as addictive to their research methodologies. The latter in particular seems to purely encourage the completion of repetitive, monotonous tasks – not something that would benefit the quality of research outputs.

9. There is plenty of scope to improve beyond the debrief

John Clay talked about ways in which researchers can improve the way that debriefs are communicated. This is an area that many (too many) researchers need to improve upon, but an even more pressing area of improvement is what occurs after the debrief.  Spencer Murrell’s presentation on insight translation covered this.

Summary slides and executive summaries are important in debriefing research, but it is important to go beyond the report/presentation into a framework that can be referenced in future. Whether it is a model that can be stuck on a wall, or a cheat sheet that can be put on a post-it note, there are many creative ways in which the core findings of a project can be transformed into an ongoing reminder. Evidently, this could easily descend into a gimmicky farce, but it is important to remember that the debrief isn’t the end of the project. In many ways, it is only the end of the beginning. The next phase – actually using that information to improve the organisation – is the most important. Any ways in which researchers can add value to this stage can only improve their standing with their clients.

10. Online conferences can work

On the whole, I think the event can be viewed as a huge success. For an affordable fee ($50), I listened to many intelligent speakers on a variety of topics, as shown by both this post and the previous post.

There was also plenty of excellent discussion around the talks and the content on Twitter, using the #NewMR hashtag. I’m usually reticent to tweet during events, but given the lack of face-to-face contact and the fact I was facing my computer at the time, Twitter worked excellently as a forum to amplify the usefulness of the content presented.

An idea is one thing, but executing it is something very different. Aside from the odd technical hitch (inevitable given the volume of speakers from across the globe), the day ran impeccably. So Ray Poynter and his board deserve huge congratulations for not only the concept, but also the organisation and output of the event. I would wholeheartedly recommend people with an interest in research investigate the New MR site and list of upcoming events.

sk

Advertisement

New MR Virtual Festival notes

I’m breaking my longest-to-date blogging absence (work-life parity should soon be restored) with two versions of the same post. This is the first.

They are related to the New MR global online conference that ran on 9^th December 2010, featuring speakers and moderators across Europe, the Americas and Asia-Pacific. The event was created and organised by Ray Poynter – a long-standing, committed and energetic member of the international market research community – and his management board. In addition to the core event, various “fringe” events also took place. More info on them can be found at the 2010 Festival pages on the site.

This is the larger of the two posts, where I’ve reformatted all of the notes I made on the day and supplemented them with some additional thoughts (I’ve not yet caught up on the presentations I missed for reasons such as being asleep/exhausted, while some presentations weren’t as relevant to me and so I skipped them). So while this isn’t exhaustive, there will still be plenty of words to keep you occupied for a short while.

I’ll follow it up with a shorter post outlining my key takeaways from the day, and my overall thoughts on the event.

As this is the single longest post on this blog (circa 5,000 words), I’m taking the rare step of putting the bulk of it behind a cut (the size also means it is not properly proof-read). Click through to continue (unless you are reading via RSS)

My name is my name

So says Marlo Stanfield. And he has a point.

Reputation means a lot. But reputation is about perception, and there are multiple perspectives in which it can be viewed.

Broadly, reputation can be thought of in four inter-related spheres

• Yourself – your personal brand
• Your organisation (this itself can have several facets, if your organisation is part of a larger conglomerate or affiliation)
• Your industry
• The wider public

Marlo is concerned with his personal reputation among people in the industry – “the game”. He isn’t so worried about the other facets.

With the prominence of polling in the upcoming general election, the research industry is contemplating its reputation among the wider public.

I don’t think it really matters.

This election is more partisan and contentious than any I recall (most likely driven by the likelihood of change, rising prominence of online media giving a voice to more people, and the novelty of the leadership debates). Pot-shots, such as those against YouGov, are inevitable. This article from Research Live shows how YouGov aren’t doing themselves any favours in their need for speed (and this is leaving aside their associations with The Sun/Murdoch/Conservative Party).

I don’t think it matters because the research industry is rarely public facing – the only publicity it really receives is through political polls and PR research.

I’ve written about the problems with PR research in the past, but there is evidently a market for it and so the method prospers. It might damage the reputation of the industry to the wider public but outside of recruitment  (of staff and respondents/participants) it isn’t really relevant.

As Marlo noted, it is industry reputation – for yourself and your organisation – that really matters.

It is similar to the advertising industry. Successful companies have a lot of brand equity through the quality and associations of their work – Wieden & Kennedy and Nike, Fallon and Cadbury, HHCL and Tango, and Crispin Porter & Bogusky and Burger King, to give but four examples.

But what proportion of the general public has heard of these companies, let alone recognises and appreciates their work? Not many. Is it a damning indictment of the strength of the marketing industry that it fails in promoting the most basic thing – itself? Not really. Companies attract talent and business through their successes and image – public perception doesn’t factor.

Ray Poynter is rightly concerned with the the ethics of market research but for me, the importance of this is in maintaining business links. There is no adequate means of policing the research industry – anyone can knock on a door and say they are doing a survey – so it is not a battle worth fighting.

Companies stand and fall by the quality of their work – or at least the perception of it within the industry. Sub-standard work that is openly criticised will only harm long-term prosperity.

Self-regulation and recognition, whether through a recognised body like the Market Research Society, or at a more ad hoc level, can achieve this through highlighting good and bad practice.The research industry needs to be more vocal in showcasing good work, and castigating poor work.

This in turn will filter to the individual level, where the talented and ambitious will compete to work for the top companies. This in turn strengthens the work, and thus the industry. It could even permeate to the public.

There is no quick fix to improve the standing of an industry, and in some cases it isn’t necessarily desirable. Rather than look to the big picture, we should focus on the more immediate challenges.

If we all concentrate on undertaking the best possible work, then a strong reputation – for ourselves, our organisation and our industry – will follow.

sk

NB: The clip of the scene with the quote is below (it is from Series 5, so beware of potential spoilers)

Facebook Polls could be pretty useful

At the recent World Economic Forum, Facebook Global Markets Director Randi Zuckerberg demonstrated Facebook polls. This, accompanied by an interview in the Telegraph, has sent the blogosphere aflutter in two separate directions.

In one corner are those excited by the prospect of 120,000 responses in 20 minutes (as a question on Barack Obama’s stimulus plan received). In the other are those concerned with online privacy and civil liberties (Given the tone of this Comment is Free article on phorm, it is surely a matter of time before the Guardian whips up a fresh batch of hysteria on the matter. And this comes from a Guardian reader).

I’m in the former category. In limited situations, it has the potential to be a valuable tool.

This is something of an about-turn since my recent rant against online polls. However, within a limited sphere of usage I can see the value. If someone wants to quickly know “what”, then this seems like a valid option. If they want to know “why”, then they should look elsewhere.

What are Facebook polls?

If my understanding is correct, a Facebook Poll is a question that appears within one’s newsfeed – similar to a sponsored ad. The user can then answer or ignore the question.

Questions are targeted to a specific audience, and basic quotas on responses can be set. Facebook has since denied that it will use personal data for Facebook Polls, but I would think that this is just semantics. Behavioural data and interests may not be used, but the poll would be pretty useless if basic age, gender and location information wasn’t monitored.

Despite this recent chatter, it should be pointed out that Facebook Polls aren’t new. Ray Poynter ran an experiment during the London Mayoral elections, and it was only last month that they were put on hiatus. This merely marks a repackaging of an existing product.

Another necessary correction is the conflation of Facebook Polls and Engagement Ads. Engagement Ads are a separate product – namely advertising widgets that can be shared and commented upon. Jeremiah Owyang has a summary here.

What are the benefits of Facebook Polls?

The nature of Facebook offers several benefits to a polling tool

• A captive audience regularly refreshing their news feed provides a fast response
• Real-time response mechanism allows immediate analysis
• Relative unobtrusiveness and simplicity could give a decent response rate
• Scale of either large response, or decent response among a targeted niche (if permitted)
• Traction among the public allows for decent tracking over time

What are the uncertainties surrounding Facebook Polls?

A rudimentary polling tool is bound to be limited. Areas that need to be explored include

• How limited will the infrastructure be? Not only in types of question, but even character limits (Shouldn’t be a problem for Twitter users)
• How attentive are the users? They are going to be multi-tasking and processing a great deal of information – how much thought will they put into an answer? (But in some cases, gut reaction is desired)
• How representative will respondents be? Not only in terms of non-Facebook population, but also with the heavy user and participation bias within Facebook (Though present research panels are hardly a beacon of quality in this respect)
• Will demographic info used for analysis be accurate? People are projecting a public persona and so may be tempted to lie on some matters. However, I cannot see age, gender or location being any more innaccurate than present survey panels
• What sort of buy-in would it receive from the industry? There is little point using Facebook Polls if no-one trusts or uses them

What is the future of Facebook Polls?

How could Facebook Polls expand? Matt Rhodes is sceptical that this will evolve into a research community. I’m less so.

Poll users can be directed to a group or page, where questions surrounding a certain topic can be explored in more depth and with more nuance. They may not evolve into a full “community”, but if visitors can be persuaded to return then they will have some value. This evolution would also support a recommendation or sharing service whereby respondents can be recruited by friends – engagement that is part of the Facebook experience (though it would have to be handled better than the various Ninjas vs Zombies widgets).

Furthermore, if these people were to opt into sharing some of their personal information then a very rich understanding of behaviour and opinion could emerge. People are on Facebook to talk and procrastinate – we should try and utilise this state of mind.

Meanwhile, looking further ahead, Read Write Web ponders the introduction of a “sentiment engine”, whereby prevailing moods and attitudes can be judged through a contextual understanding of status updates.

This would be great if a robust analytic tool could be developed, but in the meantime it would be overcome by bored people proclaiming their hunger / tiredness / hangover.

Though unproven, I can see the potential in Facebook (re)launching this product, and it will be interesting to see how it develops.

sk

Image credit: http://www.flickr.com/photos/stephenpoff/