Ten things I learned from the New MR Virtual Festival

My previous post included all of the notes I took while listening into the New MR Virtual Festival. This post contains the key things I took away from the day, and have subsequently been mulling over in the 2 months (!) since.

NB: All slides shown below are taken entirely without permission. If a creator objects to its use, please contact me and I’ll remove it.

1. The boundaries between participation and observation can (and, in some circumstances, should) be blurred

Although Ray Poynter touched on the range of methods in which research can overtly or covertly be conducted online, Netnography (cf. Robert Kozinets) is – to me – one of the more interesting. To be effective, it needs to have the research both participate and observe the environment of interest.

Erica Ruyle argues that observation (or lurking) is fine in the initial stage, since the norms and cultures need to be understood and respected. But active participation is vital in order to get than “insider” knowledge and to be able to read between the lines of the interactions.

This is a difficult proposition to promote commercially as a) the time investment (and thus cost) required will be huge and b) the researcher will need to have a personal as well as professional interest in the topic in order to both be accepted by the community and accept the community. For instance, how many researchers would turn their nose up at being asked to take part in World of Warcraft for 6 months?

Nevertheless, in the right circumstances it could prove to be a fascinating and rewarding exercise.

2. Convenience samples can still be worthwhile

It is true that all non-census research incorporates a convenience sample to some extent. But some methods require more convenience (and thus are less representative) than others.

Annelies Verhaeghe highlighted some of the issues to be aware of when conducting social media research – particularly that we should resolve ourselves to not always know who we are speaking to or monitoring.

Furthermore, something I had not considered but makes sense is that even though companies trumpet the volume of data they scrape and collect, only a sub-sample of that will be analysed due to the diminishing returns of going deeper into a very large data set.

If we’re able to augment social media research with other techniques or data sources – Annie Pettit mentioned some of the benefits of combining social media research with surveys – then it can be a very valuable and insightful method of getting real-world information on past actions and thoughts.

3. Respondents shouldn’t necessarily be viewed equally

Both Rijn Vogelaar and Mark Earls talked about premises realised more thoroughly in their books – The SuperPromoter and Herd respectively.

Segmenting the audience isn’t a new phenomenon – we often restrict our universes to who we are interested in – but within these universes perhaps we should pay more attention to some individuals more than others – particularly given the complex social interactions that cause ideas and opinions to spread. I’m not clever enough to be able to incorporate full network theories into any of my research – in the manner of Duncan Watts, for instance – but perhaps there is an argument for applying simple weights to some projects, to account for some opinions becoming more important than others. Or perhaps it is too contentious to implement without proper academic grounding and proof.

4. More of an effort needs to be made to meet respondents on their terms

Betty Adamou joined the likes of Stowe Boyd in saying that email is on the decline among younger people. This is problematic for online surveys, given that online panels are predominantly administered via email. Given the trends, perhaps we should be looking to Facebook, Twitter, instant messenger etc for both initial recruitment of these audiences and then allow them to dictate how we can contact them to alert them with new surveys. I’m not sure whether a note on a Facebook group could be as effective as an email, but it is certainly worth exploring.

5. Survey structures can do more to take advantage of the online environment

Our media and communications channels have fragmented but the content providers retain a centralised hub of controlling activity. Why can’t the research industry do this? John Dick talked through Civic Science’s modular research methodology, whereby questions are asked in chunks of two or three at a time, but combined at the back-end to build up extensive profiles of respondents.

This approach makes intuitive sense. In face to face research, the main cost was in finding people to speak to. Thus, once they were located, it was efficient to collect as much information as possible. The web is abundant with people, who are time-poor. The cost isn’t in finding them, it is keeping them. People could easily answer three questions a day if there was the possibility of a prize draw. They would be less willing to spend 30 minutes going through laborious and repetitive questions.

There are clearly downsides to this method and plenty of issues to overcome regarding data quality assurances, but the notion of Facebook users answering a couple of questions a day sounds like a feasible way to collect information among people who might be unwilling to sign up to an online survey

6. Surveys don’t have to be boring or static…

Another aspect of the online world that should be explored further is the level of interactivity. Many online surveys are straight ports of face to face surveys – a shame when there are so many more things that a web survey can – in theory – be capable of.

Jon Puleston of GMI highlighted several of their experiments in this area. Interestingly, although interactive surveys take longer, respondents are more engaged, enjoy them more and give “better” answers. I particularly like the idea of timing respondents to give as many answers as possible within a given timeframe. This appeals to people’s competitive nature, and means they’d spend far longer on it than they normally would.

The King of Shaves case study was very interesting. Rather than a 1 minute introduction for a 10 minute survey, this example reversed the process. People were given a detailed briefing on the role of a copywriter, and asked to come up with a creative slogan. In subsequent testing, seven “user-generated” ideas scored better than the advertising agency.

7. But we should be aware of the implications of survey design on data capture…

Jon’s examples showed how framing questions can improve data collection. But Bernie Malinoff warned us that even minor superficial changes to a survey can have a big impact on how people answer questions. For instance, the placement of the marker on a slider scale can heavily impact the distribution of answers.

Bernie also had some important lessons in survey usability – ranging from the wordiness of the questions (something I’ve been guilty of in the past) to the placement of error messages and how they can influence subsequent responses.

Surprisingly, his research found that survey enjoyment was comparable among people who did traditional “radio button” style surveys versus richer experiences, and that people were less willing to take part in future after having completed a flash-based survey.

It acts as a sobering counter-point to Jon’s presentation, but I inferred some caveats to this research (or perhaps I am only seeing what I want to see). I suspect some of the resistance to flash might be down to the “newness” of the survey design rather than a genuine preference for radio-button style surveys. Similarly, design iterations aren’t neutral – I wouldn’t mind different results so long as I felt they were “better” (and any methods to discourage survey cheaters are welcome). Nevertheless, it an important reminder that a better designed survey is only an improvement if it makes the survey more usable and easier to understand, and I completely agree with the final point that the industry should reinforce best practices for interface design.

8. …And whether they are suitable for the audience

Tom Ewing’s talk on gaming and research covered many interesting points, but the one that stuck with me is that it isn’t for everyone. As he points out, FourSquare holds little appeal to him (unless he wanted to be Mayor of his child’s nursery). Similarly, while the number of gamers is rising, it isn’t everyone and so we cannot assume that introducing interactive, exploratory or personalised experiences will automatically make respondents enjoy our surveys more.

Particularly since games design is pretty hard – Angry Birds and Farmville may look simple, but I wouldn’t expect any research agency to devise and incorporate something as addictive to their research methodologies. The latter in particular seems to purely encourage the completion of repetitive, monotonous tasks – not something that would benefit the quality of research outputs.

9. There is plenty of scope to improve beyond the debrief

John Clay talked about ways in which researchers can improve the way that debriefs are communicated. This is an area that many (too many) researchers need to improve upon, but an even more pressing area of improvement is what occurs after the debrief.  Spencer Murrell’s presentation on insight translation covered this.

Summary slides and executive summaries are important in debriefing research, but it is important to go beyond the report/presentation into a framework that can be referenced in future. Whether it is a model that can be stuck on a wall, or a cheat sheet that can be put on a post-it note, there are many creative ways in which the core findings of a project can be transformed into an ongoing reminder. Evidently, this could easily descend into a gimmicky farce, but it is important to remember that the debrief isn’t the end of the project. In many ways, it is only the end of the beginning. The next phase – actually using that information to improve the organisation – is the most important. Any ways in which researchers can add value to this stage can only improve their standing with their clients.

10. Online conferences can work

On the whole, I think the event can be viewed as a huge success. For an affordable fee ($50), I listened to many intelligent speakers on a variety of topics, as shown by both this post and the previous post.

There was also plenty of excellent discussion around the talks and the content on Twitter, using the #NewMR hashtag. I’m usually reticent to tweet during events, but given the lack of face-to-face contact and the fact I was facing my computer at the time, Twitter worked excellently as a forum to amplify the usefulness of the content presented.

An idea is one thing, but executing it is something very different. Aside from the odd technical hitch (inevitable given the volume of speakers from across the globe), the day ran impeccably. So Ray Poynter and his board deserve huge congratulations for not only the concept, but also the organisation and output of the event. I would wholeheartedly recommend people with an interest in research investigate the New MR site and list of upcoming events.

sk

Advertisement

New MR Virtual Festival notes

I’m breaking my longest-to-date blogging absence (work-life parity should soon be restored) with two versions of the same post. This is the first.

They are related to the New MR global online conference that ran on 9^th December 2010, featuring speakers and moderators across Europe, the Americas and Asia-Pacific. The event was created and organised by Ray Poynter – a long-standing, committed and energetic member of the international market research community – and his management board. In addition to the core event, various “fringe” events also took place. More info on them can be found at the 2010 Festival pages on the site.

This is the larger of the two posts, where I’ve reformatted all of the notes I made on the day and supplemented them with some additional thoughts (I’ve not yet caught up on the presentations I missed for reasons such as being asleep/exhausted, while some presentations weren’t as relevant to me and so I skipped them). So while this isn’t exhaustive, there will still be plenty of words to keep you occupied for a short while.

I’ll follow it up with a shorter post outlining my key takeaways from the day, and my overall thoughts on the event.

As this is the single longest post on this blog (circa 5,000 words), I’m taking the rare step of putting the bulk of it behind a cut (the size also means it is not properly proof-read). Click through to continue (unless you are reading via RSS)

Recommended Reading – 25th July 2010

The second and final group of links from the past month I recommend you click on is below:

• Northern Planner has ten great pieces of advice on how to plan if you live outside of London. The points are good advice in general, and transferable to many other things

• Adam Rifkin uses a metaphor of pandas and lobsters to suggest why Google will never be as good as Facebook when it comes to social applications

• Neil Perkin presents the ten key principles behind digital community

• Tom Ewing uses a dalek metaphor to explain why research communities shouldn’t be too concerned with presenting a natural environment

• Ana Andjelic believes that strategy is becoming hard to plan and implement in our increasingly complex environment

• Leo Hickman in the Guardian has a great article on some of the privacy issues surrounding the use of Foursquare and other social networks

• Gregg Fraley has a great article containing tips on how to get the most from brainstorming sessions

sk

Recommending Reading – 4th April 2010

If you have a spare 15 minutes this weekend, you could do worse than read the following:

Jonah Lehrer looks at Costco through his Neuroscience prism. I’m not quite sure it adequately explains why people choose to pay a subscription to enter the store, but it is still interesting reading.

Two sides of a similar coin: Tom Ewing’s latest Poptimist post considers the options of a Foursquare for music, while Paul Lamere considers a Last.fm for books. The underlying point – the more metadata collected and used, the better!

Clay Shirky’s blog postings are sparse, but always incredibly valuable. This preview of his next book – on cognitive surplus – is no exception. It prompted Kevin Kelly to announce the Shirky Principle – complex solutions  can become so dedicated to the problem they are the solution to, that often they inadvertently perpetuate the problem

Diane Hessan has some provocative thoughts on the next generation of market research. As always, the solution depends on the problem, but I liked the points made.

This link in particular is a hard sell, but I wholeheartedly recommend you read why Bill Simmons has fallen back in love with Sabermetrics. Advanced baseball stats may not be to everyone’s taste, but it shows the power and beauty of numbers. For a more gentle introduction, you might want to read my review of Michael Lewis’ Moneyball, in which Simmons heavily features.

sk

Recommended reading – March 19th 2010

As well as reading material, I also recommend viewing this 5 minute video on a near future social media storm, which is extremely well done:

Mob (a near-future science fiction story) by Tom Scott from hurryonhome on Vimeo.

In addition, I would recommend reading:

Neil Perkin has a fantastic overview of the case for agile planning, and why businesses need to be able to move quick in the current marketplace

Ben Kunz has pulled together the highlights of danah boyd’s recent SXSW talk on privacy in a highly readable manner

In a typically entertaining post, Rory Sutherland makes an oblique case for creativity, and on why the things that can be measured easily aren’t necessarily the best. He also gives props to the Midlands

I liked this Boston Globe piece on Mesofacts – things that change the same, albeit slowly. Things like populations or incidence rates. It is a useful reminder to question the basis for our assumptions

And, finally, Tom Ewing has a preview of his forthcoming MRS Conference paper, where he characterises the “networked web” experience. I liked the point on how the dynamism of the structure works alongside the dynamism of the content.

sk

How can research inspire?

The question in the title is predicated on the assumption that research can inspire. While the haters may disagree, I truly believe it can.

Understanding the different ways in which it can do so is trickier.

In a slight contradiction to my previous post on “insight”, I’m using the term “research in its most catch-all form. Rather than restricting the thinking to groups or surveys, I’m thinking about all disciplines and all methodologies. Research, data and insight.

In order for research to inspire, the recipient needs to be able to be inspired. Some form of creative process in order to make that new connection or leap is necessary.

In thinking about how research can inspire, I’ve come up with three initial ways. It is by no means a typology and the examples aren’t even mutually exclusive but it seems like a good start in which to organise my thoughts.

Structure:

The way in which research issues are approached and the problems framed. Examples include:

• Methodology: The methodology itself could suggest new and previously alien ways to approach an issue. This post from Tom Ewing highlights some innovations in how research is carried out, but there are numerous examples of fresh approaches – from fMRI scanning to crowdsourcing.
• Influences: Research is often (correctly) portrayed as insular but there are notable exceptions – Tom Ewing himself being one of them. He is able to take his knowledge and skills from music criticism and community building and apply them to research problems. Admittedly, this example isn’t research-specific but it nevertheless can inspire others to bring in people with different perspectives
• Backwards approach: I mean this in a good way – research briefs are often issued to answer specific questions. To discover the most relevant way to get this information, researchers need to start with the answer and work backwards to figure out both the question and the way in which it is asked

Results

While a lot of research may be characterised otherwise, results themselves can inspire:

• Exploratory research: By its very nature is designed to uncover new observations or – deep breath – insights
• Fresh perspectives: Seeking to understand different audiences can lead to fresh outlooks as we look at the same issue from someone else’s eyes. While the Morgan Stanley note from their 15 year old intern was undoubtedly overplayed, I did like the notion that teenagers stay away from Twitter because it is full of old people trying to be young (for what it’s worth, I view Twitter as being far closer to Linked In than Facebook – it is useful connections rather than genuine relationships)
• Holistic understanding: On a larger scale, ethnographers like Jan Chipchase offer us fascinating observations into areas we would never have even previously considered
• Prototyping: I’ve written about IDEO before, and I love how they actually physically build things in order to better understand the problems
• Desk research: Somewhat tenuous, but even sitting at your desk and reading, and being inspired, by different blogs or sites can be considered a form of research – whether one is explicitly looking for specific information or not

Implementation and Impact

Moving on from the results themselves, how research is used or the effects it has may also inspire

• Workshops: Debating how research can be used can lead to further thoughts on idea implementation
• Social effects of making data public: From last.fm to Nike+ making personal data available both encourages further participation and causes people to adjust their natural behaviour
• Rewards and recognition: Similarly, in communities there have been noticeable effects on user behaviour and community culture when elements such as post counts or social connections have been introduced
• Analytics: Avainash Kaushik is a Google Analytics evangelist who is full of great examples in how understanding site data has improved business performance

This question was recently posed to me by a colleague working on an assignment. The assignment is ongoing so any further thoughts, ideas or examples on how research methods, results or implementation can inspire would be massively appreciated.

And perhaps this attempt at crowdsourcing opinion will inspire others to a solution for the issues they are facing…

sk

Image credit: http://www.flickr.com/photos/stephenpoff/

Links – 8th March 2009

My recommendations for the past week include:

• Paul Graham on why he thinks social media has contributed to the death of TV. He makes some good points on the social nature of TV, but I disagree that synchronicity will fade away. TV will continue to prompt watercooler chat around shared experiences. If the watercooler is the workplace, then a show only needs to be viewed the previous evening – not necessarily at the same time. But if the watercooler is Facebook or Twitter, then synchronicity and real-time feedback still matter.

• Penelope Truck thinks we shouldn’t look to grad school as a way of mitigating the effects of the recession

• Adrian Ho looks at the role culture plays in companies and self-organised groups

• Farhad Manjoo doesn’t see Twitter killing Google nor Facebook

• Al Ries writes that consumers only love brands once they know them. In a competitive market, familiarity can be a barrier to switching

• Jeremiah Owyang proposes that companies should look to the social web for opt-in consumer information, which would remove the need for registration forms

• Tom Ewing explores the differences in rating music and films, and wonders what effects this has on accumulator sites such as Metacritic

• A recent study suggests that leaders emerge by talking first and often, but not necessarily making the best points

• And I’ve been adding some great Slideshare presentations to my favourites

sk

Links – 22nd February 2009

Some of the things I’ve read over the past week and would recommend:

• There have been some interesting events occurring in London over the past week: John Griffiths went to the NESTA workshop, and Anjali Ramachandran and Eaon Pritchard both went to see Seth Godin speak

• A thought-provoking article in the Atlantic on the future of TV. It argues that TV’s USP is immediacy. While there are still cultural reference points via TV, scripted shows will increasingly see TV as just another distribution pattern. TV will therefore move to concentrate on news, current affairs, live reality shows and sport. This makes sense to me given my research – TV excels at events which are essentially DTR-proof, and the most popular shows online are dramas and comedies that can be viewed at leisure and shared/discussed asynchronously. However, I would argue that successful scripted shows still need TV as that anchor point for mainstream cultural crossover.

• Another article on the future of TV, this time in Advertising Age, comes from the perspective of consumer choice in the ad-supported model

• A fascinating post in Salon on Nicholas Nassim Taleb and our weaknesses in risk assessment

• Ana Andjelic has a great post on our general failure to accurately predict the future. Not only does she argue that a lot of campaigns will fail, but also that our limited perspective means we will often follow the same patterns (potentially of failure)

• Tom Ewing takes a line from a Bob Dylan track to point to a problem faced by market research and information consultancy

• Nigel Hollis asks if retail brands are inherently local

• A discussion with Rishad Tobaccowala, Faris Yakob and David Pescovitz on the future of technology and marketing produces some great soundbites

• Writing in Mashable, Whitney Hess gives a nice overview of user experience design

• Business Week seeks to debunk six social media myths

• Caleb Kramer’s lessons learned from brandjacking Burger King on Twitter

sk