New MR Virtual Festival notes

I’m breaking my longest-to-date blogging absence (work-life parity should soon be restored) with two versions of the same post. This is the first.

They are related to the New MR global online conference that ran on 9^th December 2010, featuring speakers and moderators across Europe, the Americas and Asia-Pacific. The event was created and organised by Ray Poynter – a long-standing, committed and energetic member of the international market research community – and his management board. In addition to the core event, various “fringe” events also took place. More info on them can be found at the 2010 Festival pages on the site.

This is the larger of the two posts, where I’ve reformatted all of the notes I made on the day and supplemented them with some additional thoughts (I’ve not yet caught up on the presentations I missed for reasons such as being asleep/exhausted, while some presentations weren’t as relevant to me and so I skipped them). So while this isn’t exhaustive, there will still be plenty of words to keep you occupied for a short while.

I’ll follow it up with a shorter post outlining my key takeaways from the day, and my overall thoughts on the event.

As this is the single longest post on this blog (circa 5,000 words), I’m taking the rare step of putting the bulk of it behind a cut (the size also means it is not properly proof-read). Click through to continue (unless you are reading via RSS)

Clash of the Titans

Prior to the main event, there was a debate, hosted by BaQMaR and moderated by Joanna Chrzanowska, on whether online or in-person qualitative research was the better methodology.

Three topics were debated, with three people on each side of the debate. Representing the onliners were Tom de Ruyck (Insites), Andreiko Kerdemelidis (Visions Live) and Julie Wittes Schlack (Communispace). Debating the in-person side were Dr Eleanor Atton (Razor Research), Simon Patterson (Qri Consulting) and Geoff Bayley (Independent).

The debate was never really on an even battlefield, as the terms of reference kept shifting. Different methods have different strengths and weaknesses – whether traditional focus groups, bulletin boards, longitudinal ethnography or passive data mining. At different points, different methods were cited to defend online or face-to-face.

Beyond the logistical arguments over speed and cost, things I noted include:

In favour of online

• It gives clients a direct link to either interact directly or observe participants
• It is more scalable, so more people in more categories/segments can be covered
• More tools and applications are available for participants to be able to express themselves
• The most interesting part of a focus group is the last half hour – but you get that throughout online research as the relationship already exists

In favour of face-to-face

• Online can be used to supplement face to face (e.g. pre/post tasks) but the personal relationship needs to be core
• There are more identity cues and emotions are easier to detect
• The speed and spontaneity of discussion is greater when talking than when typing – in the energy of the moment there is scope for self-discovery
• You need to meet someone in person to get a true representation of their world and go beyond things they are initially prepared to tell you

We were told afterwards that the in-person debaters had won – not because they had a majority but because a higher percentage voted in their favour after the debate than had done beforehand. Given the bias of debating online to an online audience, I suppose this was the fairest way of adjudicating.

New MR Main Stage Part 1 – Asia-Pacific, Session 1

Chaired by Jeffrey Henning (Vovici) and convened by Greg Coops (Asian Strategies – and not to be confused with this guy). I was planning to listen to a couple of sessions before I went to bed but with Alistair Gordon’s technical difficulties, I only listened to one talk:

Ray Poynter (The Future Place) – “How can we turn online discussion into insight”

1am UK time was perhaps not the optimal time to be listening to Foucault references, but from what I gather Foucault distinguished language (what we say) and practice (what we do), with discourse connecting the two. This discourse creates meaning – for example, everything we know about brands is created from the discourse around it. Online discourse can improve our interpretation of this.

There are four steps to insight, according to Ray

• Understand the client’s needs
• Find the discourse
• Analyse the discourse
• Synthesise the needs and analysis (which is the reason we get paid)

And there are three main strategies to achieve this

• Netnography (cf Robert V. Kozinets) – he takes a hardline position on netnography in that it has to both participation/observer rather than just participant (covert) or observer (behind the glass). There are a range of analysis options include hermeneutics, semiotics, discourse analysis and NLP
• Create discussions e.g. My Starbucks Idea. This only scratches the surface as you need to create conversations, but there is also an ethical conundrum as you normally wouldn’t want to feed research conversations into marketing, yet you can use marketing conversations to feed into research
• Mass techniques – looking at volume, trends, sentiment, influence and content through monitoring and analysis. There are more ethical concerns about public vs private information (and the known vs unknown aspects of that) as well as issues regarding marketing (brands will want to know who the influencers are) and in confidence in results (since they can’t be tested for statistical significance).

Ray concluded that software is getting better at finding information (although only 15% of the web is searchable and only 15% of that is searched), but people are still needed for analysis, though it will continue to be the case that people will disagree over the interpretation.

New MR Main Stage Part 2 – Europe, Session 1

Chaired by Dan Foreman (Active Group) and convened by Sue York (The Future Place). I initially overslept

I managed to oversleep and miss the presentation on Digividuals from John Kearon of Brainjuicer. Given that it was voted the second best paper of the day, I really should catch up on it.

Tom Ewing (Kantar Operations) – Surveys are Videogames

Tom’s talk on “gamification” was one of cautious optimism – it can work, and it some ways it already does, but there are many dangers that need to be overcome.

According to Tom, surveys are quite like videogames as they are both artificial environments built for players to progress through, sustaining and rewarding their interest. They both create narrative exoskeletons rather than tell stories, while cheats can also prosper.

However, the key difference between the two is that games build worlds and encourage people to build strategies to finish, while research discourages strategies and masks the environment.

If we are looking to game mechanics to improve surveys, we have to remember that the mechanics are all that can be controlled – not the gameplay itself (the example used was Tumblr’s Tumblarity, that ended up rewarding junk rather than rich community). Players respond to prompts in different ways, and indeed will prefer different ways of navigating games. Some will prefer to get through the game as quickly as possible, others will seek to attain perfection on one level before progressing to the next.

Remembering that people are different also raises questions about authenticity and how representative the audience are. For instance, Foursquare is by young people for young people. We have to understand who our audience is before we can interrupt and transform their world.

Another note of caution is that gamification shouldn’t be pointsification – it is not just about collecting points, but about understanding the addictive nature of good games. We should avoid rewarding people for routine, monotonous tasks (I’ve already asked if Farmville is the equivalent of a survey) and instead look to build on curiosity and excitement.

Tom finally pointed out that we can look towards mobile and casual games for inspiration – simple, modular activities that can be dipped in and out of, rather than complex communities that require constant attention.

Overall, a very interesting presenting that has some ramifications for how communities are built and maintained.

Jon Puleston (GMI Interactive) – The Eureka Experiments: Exploring ways to stimulate and evaluate creative ideas in online surveys

Not only did I find Jon’s presentation interesting as well, but some of the implications of his research is highly relevant to both me and others who design and run online surveys. Surveys are gradually starting to look a bit nicer (and not just ports of pen and paper surveys), but this paper points to some of the ways in which research can develop and iterate further.

Jon has found that ways to improve feedback in online surveys include

• A clear, visual introduction
• Relevant imagery
• Projective techniques (NOTE: I’ve not had much success with this, but then it will depend on the nature and execution)
• Humour
• Showing other “good” responses
• Introducing some competition (e.g. a countdown clock)
• The right wording and stimulus, for more accurate framing

These techniques are not just a question of engagement, but they also help to unlock the thinking process, which in turn facilitates creative testing and less rote feedback

There have been three stages to Jon’s experiments thus far:

• Icebreakers are needed to get people in the right frame of mind and right thinking process. These can include name games, writing lyrics, imagining silly uses for a pen. It may only produce 6-8 ideas but it unlocks some potential radical thinking and highlights that it isn’t a “normal” survey
• A different briefing process: For a King of Shaves project, the survey had a 10 minute briefing process explaining the basic techniques of advertising development, which was then followed by a couple of questions. 250 respondents generated 1200 ideas, with 87% cross-participating in follow-up evaluation surveys. 8 of the ideas outscored those from the agency.
• Exploring specific idea stimulation techniques – such as thinking rooms for people to freely explore, enquiring what sort of deodorant Apple would produce, or replying to an idea with another idea to then further prompt people

New MR Main Stage Part 2 – Europe, Session 2

Chaired by Brian Jacobs (BJ&A) and convened by Sue York (The Future Place).

Annelies Verhaeghe (InSites Consulting) – Beyond the Hype: The Entrance of Social Media

According to Annelies, the content of social media has long been denied, when there is in fact a lot of valuable information that can be found on it. She said that conversation is the new unit of analysis (though to be honest, I’m not quite sure what she meant by this).

However, a few things to be aware of

• We  need to largely give up on the idea of a representative sample since we often don’t know who we are speaking to or monitoring
• Sampling is still important, as only a small proportion of the information scraped will be used. It might be a convenience sample, but additional data from elsewhere should be sought if available
• Data quality checks need to take into account block quotes in replies, and spambots, in order to get a true reflection
• You need to understand whether your goals are around activation, evaluation, message or brand fit
• You may only end up learning what you already know, if you stick to keyword analysis

As an example of outputs, she showed a matrix that plotted volume of buzz against sentiment. Their way of calculating sentiment is based on emotionality (level of emotion in comment), proportion of positive comments, proportion of negative comments and average rating review sites.

Rijn Vogelaar – Introducing the superpromoter

Rijn’s talk was based on his book. His premise is that we are shifting from individual opinions to interhuman behaviour, moving from averages to extremes (the bell curve to power laws) and that the subconscious and emotions are becoming more important.

Therefore superpromoters – people that are enthusiastic, share that enthusiasm and influence others, are more important than the average consumer and should be focused upon. They cause growth in revenue and reputation and are great motivators and ideal co-creators. Companies should spend 80% of their time focusing on superpromoters since they need to work together to be successful. These people would be critical of the brand, as a friend would be

However, the antipromoter should also be studied (presumably with the other 20% of time) – they are not only dissatisfied but are outspokenly negative, share that and also have influence. We are often blind to this, as we assume enthusiastic people are loyal and the focus is on acquisition and incremental improvement.

Listening to this, I took the point that the most successful brands are often divisive, but I wasn’t entirely convinced by his assertion that even energy companies will have superpromoters. As different companies will have different types of superpromoters, they may also cancel each other out as they try to influence the vast number of people who are indifferent (presumably the people who Rijn thinks we should overlook, even though they would make up the majority of sales).

Richard Shaw (Virtual Surveys) – Hacking the Data Shadow

The hacker ethic is balanced between creativity and passion, money and motivation, and self-determined used of time. Combined, these encourage playfulness and experimentation with problem-solving in small, iterative steps. This level of self-determinism can come up with some quite innovative solutions. For instance, Rich gave the example that he spent a few days creating an iPhone app, that would allow people to record their consumption habits when in pubs. A cheap and elegant solution that will produce more accurate data (among people with iPhones) than a survey.

New MR Main Stage Part 2 – Europe, Session 3

Chaired by Brian Jacobs (BJ&A) and convened by Sue York (The Future Place).

Dr Agnes Nairn (EM Lyon Business School) – Ethics for new MR

Agnes talked about how the MRS relies on the confidence that research is conducted honestly, objectively and without intrusion or skews.

The three cores of market research ethics are voluntarily, informed consent (people aren’t forced, they understand, and they can refuse). However, in addition, there is also privacy – the right to be left alone.

With children online, they often don’t realise the implications of what they are posting and can later come to regret it. However, that information is still available to be scraped and used. Kids online are also used to lying about their age – if children are recruited from an internet service for research, age needs to be confirmed by a parent over the phone. Agnes also pointed out that if some incentives are too large, it takes away the voluntary part of research since it is effectively bribery (something I don’t agree with).

Other issues touched upon include sugging, transparency, professional respondents and the digital divide.

Ian Ralph (GfK NOP) – Embracing the 3rd screen

This paper looked at the differences in data capture with mobile versus a computer. The study found that on a mobile the response length was as good, with more frequent responses, although there were more typos and the difficulties with the screen size and keyboard made it more difficult for conversations. There are also issues to consider around battery and 3G connectivity.

The mobile should be seen as a window to the PC – although behaviour on devices differs, functionality should be mirrored. Phone and computer can be used in conjunction on some projects – the computer acts as the hub with more in-depth tasks completed, while the mobile can be used for quick, time or place sensitive information. One of the mobile’s strengths is that it is in the final 5 yards of the consumer experience. It should be seen as a technology, not a methodology.

Bernie Malinoff (Element 54) – Sexy Questions, Dangerous results

Bernie started by making the point that, unlike with face to face or telephone research, usability matters.

Ways are being explored in making surveys more usable, to increase engagement among respondents. But this needs to be balanced with data consistency. Using a 10 point scale question, the data returned can be very different if the scale is anchored to the left, to the right, in the middle or even if using an image instead of an arrow. Even on a standard NPS question, people tend to use it more as a guide than a normative way of asking questions, and so data differs.

It has to be remembered that any changes to the design of a survey will therefore change its results.

Eye tracker information has also been used to investigate issues with surveys. We need to understand how people fill in surveys so that we can minimise these errors in future. Issues include

• Respondents advancing through surveys without properly reading the questions
• Having to go “below the fold” for responses – horizontal scales are therefore better as they don’t encounter as many screen size issues
• Error messages need to be without bias – people will gravitate towards where the error message is, indicating their response is different to what it would initially have been
• More visually engaging surveys aren’t necessarily the answer, as they can confuse people and make them less likely to participate in future

As such, as an industry we need to devise and reinforce best practices for online research interfaces, to ensure a decent minimum standard of respondent experience and data collection

Betty Adamou (Nebu) – “I’ll Facebook u yeah?” Modern Communication and evolving with the times

Betty’s entertaining presentation posited that current online research techniques aren’t relevant to young audiences

• Twentysomethings aren’t going to take half an hour out of their workday, when they don’t have much of a lunch break, to a do a survey
• Particularly when they have low attention spans
• Email isn’t used as much out of work compared to Facebook, SMS or instant messenger
• Survey language is too formal and isn’t relevant to this audience

Clearly, these points are exaggerated for effect (Sample of 1 – I’m a twenty something who prefers email to Facebook) but it is absolutely right to say that we need to identify different groups of people’s preferred means of communication. She says that email is a bit like a HiFi – you don’t want to get rid of it as you have an investment and attachment, but you rarely use it nowadays.

Her four point guide to future communications is

• If you want my attention, grab it as I’m not coming to you
• Create an app, otherwise our paths won’t cross
• Speak the language for the right age and culture. What does nowadays mean anyway?
• Do research on research. Repeating falsehoods doesn’t make them true, but people more likely to believe them

New MR Main Stage Part 3 – Americas, Session 1

Chaired by Zoe Dowling (Added Value) and convened by Andrew Jeavons (Mass Cognition)

Mark Earls (Herd Consulting) – “I’ll have what she’s having”

Mark Earls spoke on his Herd theory, whereby we should look not at individual choice (despite the Anglo-Saxon culture of independence) but on social learning. Our we-culture sets us humans apart – we have more complex social dynamics, such as copying baby names or mimicking social group clothing norms.

This means that in research we need to find the space between groups of people. We should allow people to see each other and build social recommendations to feed into individual actions, in the manner of Trip Advisor or Amazon. Each Amazon page has at least 16 pages that take individual data and flip it over to group

John Clay (Research4) – Why are market researchers so !x!?ing bad at communicating research and insights?

John’s presentation focused on something I think my company (Essential Research) is pretty good at, but that the research industry as a whole isn’t – clear and decisive communication.

He opened by talking about Prism Brain Mapping, and how it splits the brain into 4 quadrants – stability, drive, expression and analysis – to illustrate how we behave in different situations. Each segment has certain dominating characteristics, and different people are more biased to a different quadrant.

Jon contends that the research is biased towards analysis, a sales director towards drive and a marketing manager towards expressions. Researchers need to recognise clients’ (stereotypical) personalities as well as their needs, and adjust communications accordingly.

The three key aspects of communications design are

• Simplicity – we should “master the art of exclusion” to digest findings into a topline summary with bitesize actions
• Imagery – this should be both visual and spoken. Najir, LJ (1998) says we recall 10% of what we hear, 35% of what we see but 65% of what we hear and see
• A story – structuring around a central story will create cohesion. Stories get empathy and dopamine running, but the important stuff should be up front in case people need to leave halfway through). According to John, a one hour story requires thirty hours of preparation – indicating we are severely undercharging for presentations)

Rather than restrict ourselves to PowerPoint, we should use other aids such as a whiteboard or Evernote to distil things and help our thinking

Spencer Murrell ( Lextant) – Insight Translation – Turning insight into action

Spencer advocated the use of information models as insight translation, rather than written reports. Traditional research structures do the planning and doing well, but come up short in the understanding. Insights aren’t the final stage of a project – they need to be translated in order to demonstrate how they can change the business.

I generally dislike the overuse of the word “insight” but at least Spencer defined his interpretation – it should contain emotion, feature, benefit and attribute. Its principles are that it is

• Actionable – simple, concrete, sensory, cognitive
• Meaningful – directly tied to the data
• Aspirational – describes the ideal future
• Inspirational – descriptive but not prescriptive. It may have a verb, but let the client decide what that is

He advocates a research report of no more than 10 pages – not everything that was found out; just the important stuff. The rest can go in the appendix.

New MR Main Stage Part 3 – Americas, Session 2

Chaired by Annie Pettit (Conversition Strategies) and convened by Andrew Jeavons (Mass Cognition)

Diane Hessan (Communispace) – The 8 myths about successful research communities

I was expecting Diane’s talk to be similar a paper I’ve previously discussed on this blog, so I consciously limited my note-taking. Nevertheless, I still wrote down quite a few things, indicating the quality of the talk. Subsequently, Diane’s talk was voted the best of the conference, so I clearly wasn’t alone in enjoying it.

Some of Diane’s advice from having “made more community mistakes than anyone in the world” include

• The open nature of communities mean that they “can provide answers to questions we forgot to ask”
• Communispace members tend to engage with two research projects a week
• Communities are appropriate for NPD, brand and product feedback and as a market research aid (e.g. developing hypotheses or guiding pre-testing) but not for u&a, segmentation, brand equity or customer satisfaction tracking
• Bigger is not always better for communities as the objective is not the search but to understand – moderators need to engage and respond. 3-500 members is optimal, depending on the type of community
• Is it better to have 3500 contributions a month from 350,000 visits on a public site or from a 400 person community (assuming 55% participation and 16 contributions per member)?
• Technology such as filtering and auto analysis helps but it is no substitute for speaking to members appropriately
• Communities can be among casual or non-consumers, but there needs to be a focal point in terms of how the category fits into their lives
• Respondents are paid $10 per month if they meet minimum contribution requirements. Participation doesn’t go up when incentives increase. The main reason for contribution is “continuous, frequent and loud contribution to future of company”
• The best method for recruitment is online – partners include panel companies, where they recruit in very detailed ways. They don’t have own names, but put right message in email with right value proposition.

John Dick (CivicScience Inc) – One Question at a Time: Engaging Today’s Bandwidth-Strapped Consumer in High-Yield Attitudinal Research

John’s research findings come from studying 52m polls by 2.5m consumers (mostly in the US). He advocates quick, disjointed polls that can later be pulled together to provide fuller information on a respondent.

A benefit of this is that it is convenient for the respondent – the survey goes to them (e.g. through an app or Facebook advertising) rather than them having to come to it.

Typical polls are three questions at less than 200 characters per question, taking under 30 seconds in total.

Cookies, Facebook ID and IP address are used to match people across surveys and append answers together longitudinally – it is interesting to see topics spread across multiple websites as answers can provide unexpected correlations. Facebook Connect can help overcome the issue of multiple people using the same machine or deleting cookies

Large-scale profiles across a large number of questions can be made. Benefits John cites include

• Polls can generate millions of responses a day
• It works for harder to reach people (though presumably it would take time to build up their profile and understand if they were hard to reach or not)
• Low cost broad-based data collection works well for pay per performance
• It generates fast response to new online ad campaigns – data can be cut immediately, or you can find out the type of person that likes it and concentrate on them

Although the presentation was very much a sales pitch, it was all new information to me and so I did find it interesting.

New MR Main Stage Part 3 – Americas, Session 1

Chaired by Susan Sweet (Hall & Partners) and convened by Andrew Jeavons (Mass Cognition)

Jen Drolet and Adam Rosso (iModerate) – You Can Let Go of Your Focus Groups… Really it’s O.K.

Jen and Adam’s presentation sought to overcome myths of focus groups – in that it is not always best to be in person (such as with sensitive subjects), stimulus doesn’t have to be touched, and that clients can still “sit behind the glass” online.

They also pointed out the benefits of asychronicity and that online bulletin boards are good for shy people, whereas focus groups work better with extroverts.

Erica Ruyle (Market Strategies International) – Stalker for hire – How to be an Engaging Ethnographer No Matter Where You Are

Ethnography consists of both participation and observation – we cannot get to the root of issues just by viewing people. Whether in a MMORPG or in person, we need to get involved. Disimpassioned observation online is what Erica calls lurkography, or “social acceptable stalking”.

Lurking is acceptable in the first stage, in order to get used to the norms and culture of the group, but researchers should become active participants. Getting more insider knowledge leads to more accurate research and so you should strive to get as close to the activity as possible. The community should be embraced to gain a rapport (I assume disclosure is necessary).

There also needs to be the right balance of time – realistically one can’t be on community 24/7 but there is a large commitment to remain involved to ensure the conversation doesn’t go on without you. This can require quick thinking and reflexivity while acting with consistency to react to the specific context

Everything online should be recorded – whether videogames or chat logs – and complemented with handwritten notes. Different formats have their individual benefits for collecting and keeping information. However, there is no canonical way to analyse the data (as with all methodologies), though patterns and implicit connections are obviously important. The sooner you gain the trust of members, the sooner you can go beyond skew issues

Annie Pettit (Conversition Strategies) – Mr Survey Flirts with Ms Social Media Research

Annie concluded the day with a short story (a fable?) on how surveys and social media research can work together. Surveys have the quant precision and census generalisability but social media research can provide raw volumes and serendipity through unanticipated connections. Not only can these separate benefits be combined, but social media scraping can help populate quant survey lists and sentiment analysis can provide a long tail of information to complement survey data.

If you are still reading this far down, you’re either a hardy soul or you’ve skipped chunks. Irrespective, I’ll be following up this post (eventually) with the major things I took away from the presentations, and my overall thoughts of the event.

sk

Advertisement