Notes from MRG Conference 2011

A couple of weeks ago I took part in a short session at the 2011 Media Research Group Conference, which took place in London. I took some notes during the day (mainly with the earlier speakers). They are below and in chronological order, though firstly a quick exec summary:

The four papers I enjoyed most were

• Tim Harford’s keynote on life in a complex world
• Sky and Cranfield University’s examination of engagement, presented by Becky McQuade and Anne Mollen
• Claire McAlpine of Mediacom’s overview of behavioural economics
• Richard Curling of YouTube on skippable pre-rolls

Synthesising these talks, my key take-aways were:

• Run lots of prototypes and versions
• Ask audiences what they think, rather than just infer from behaviour
• Set up the tests in such a way to drive people towards the behaviours/answers you desire
• Be aware of contextual reasons that might provide counter-intuitive answers

And now for the detail…

 

Tim Harford – Problem Solving In a Complex World

Tim Harford, author of books such as The Undercover Economist) , initially walked through examples of problem solving such as

• Archie Cochrane – a Prisoner of War who conducted experiments to find out what was making people ill in the camp
• Thomas Thwaites – a student who took 9 months and spent over £1,000 to try and make a toaster from scratch and even when cheating largely failed
• Cesar Hildago – who has mapped 5,000 product categories. But Wal-Mart has 100,000 types of product in a store, and in New York there are probably 10bn

His point was around the God Complex – the conviction that no matter how complex something is or how little data is available, you know the answer. It is dangerous and yet you see it everywhere.

We need to step away from the god complex as we can’t solve things in one step. Instead, we gradually learn over time through trial and error.

For instance, Unilever wanted to create a new nozzle through for their detergent production. They hired a mathematician who failed to sufficiently improve it. Instead, they created ten random computer generated models and picked the best. They then created ten variations of this. They repeated this process twenty times. Ultimately the nozzle was much improved, although they don’t know why.

Business successes are random processes – there is no silver bullet for the perfect CEO or strategy. However, instilling a start-up culture allows experimentation to see what is best. Google has a target failure rate of 80%, but this failure has to be quick, rather than being too big to fail. In order to do this, we have to overcome loss aversion.

In the BBC documentary about Fermat’s last theorem,  Goro Shimura said in reference to his colleague Yutaka Taniyama :

Taniyama was not a very careful person as a mathematician. He made a lot of mistakes, but he made mistakes in a good direction so eventually he got the right answers. I tried to imitate him but I found out that it is very difficult to make good mistakes.

Tim fielded a couple of questions relating to popular business books

• Tom Peters’ In Search of Excellence profiled many companies to see what made them successful, but three years after the book was published around one third of them were in trouble (e.g. Wang, Atari). Were they actually excellent, or is excellence fleeting?
• James Surowiecki’s Wisdom of Crowds is often misunderstood as he himself said that it only works in specific situations – when expert judgement is no help and where the crowd can be polled independently (Duncan Watts has shown how randomness becomes important when things are dependent

Claire McAlpine – Mediacom – How are you integrating behavioural economic thinking into your work?

Inspired by thinkers such as Steven Johnson (Where Good Ideas Come From) and Chip & Dan Heath in addition to Thaler & Sunstein etc.

Hunches are where we collide ideas – these could be our ideas over time, or our ideas with other people’s. For instance, the Gutenberg printing press was inspired by the wine press.

We need to overcome cognitive biases (such as picking the second cheapest wine on the list) and recognise things such as information deficit and availability bias. We are more Homer Simpson than Spock – we are not rational agents. We may have good intentions but these can quickly be forgotten if we are in a “hot state”.

There are three stages to integrating behavioural economics

• Identifying the behavioural context
• Identifying the behavioural journey
• Identifying choice context and ultimately creating choice architecture

Claire gave the example of Special Constable recruitment. By identifying two choice contexts – career and inspiration – Mediacom were able to frame their media strategy (both in terms of creative and placement) for two separate audiences

By understanding how behaviours differ, we can seek out how to encourage the desirable ones to be replicated. The ultimate goal is to be able to switch the default behaviour, which we often resort to as a mental shortcut.

 

Mark Barber (RAB) and Jamie Allsopp (Sparkler) – Media & the Mood of the Nation

Mark and Jamie went through the research findings of this research which covered 3,500 smartphone survey responses from 1,000 people, qualitative depth interviews and diaries and EEG brain scan experiments.

The research came about from the general move in advertising from systematic (logical) to heuristic (emotional) processing, and observations that advertising works better in mood-enhancing environments.

The findings were framed using James Russell’s Circumplex Model of Affect, which places results on two -5 to +5 scales of arousal (energy) and valence (happiness).

Radio was compared to both TV and online. While all displayed rises in happiness and energy, radio showed the highest average increases in total and across the most dayparts. While this may be caused by other activities people are doing while they listen to the radio, it nevertheless means that people are in a more receptive frame of mind when it comes to processing advertising messages.

 

Becky McQuade (Sky) and Anne Mollen (Cranfield School of Management) – Online Engagment: We might be getting there

Anne said that there are two schools of thought with engagement

• It is bankrupt as it is not a metric since it is too abstract and not credible (unlike retention and acquisition)
• It is viable (she is in this camp)

The academic studies in this area have been focused on perceived interactivity and telepresence (her paper is here), but it hasn’t as yet properly been joined up to commercial requirements.

Her definition of engagement is “cognitive and affective commitment to an active relationship” and requires

• Utility/relevance
• Pleasure/enjoyment
• Dynamic and sustained cognitive process

Using Survey Interactive, they ran an online pop-up survey with 60 engagement statements (reduced from an original list of 150) on 12 point scales across 14 Sky websites (and on a NetMums panel), resulting in over 12,000 responses. This found four drivers of correlation. From the largest to smallest, these are:

• Cognitive processing e.g. enjoyment
• Temporal needs e.g. hedonic and utilitarian value (what we need and want)
• Self-congruence (identity with the brand)
• Social identity (context, environment, peer to peer communication)

Conversely, engagement isn’t

• A measure of human behaviour – there was low correlation between engagement and time spent, frequency and recency
• Behavioural footprints (actions such as subscriptions or likes) – there was only a small positive correlation among a subset of those engaged
• Activism (such as loyalty) – engagement is context dependent and not a behavioural type

The study was specific to advertising, and found those engaged had higher ad recall, improved core message delivery, more favourable opinions towards the brand and a higher likelihood to purchase (but not higher purchase intent).

Becky and Anne closed by saying for engagement to be viable it has to have a close relation to ROI and KPIs. Their NetMums study showed engagement has an impact on trust, satisfaction, loyalty and add responsiveness and has a high positive correlation with the Net Promoter Score.

Anne isn’t linked exclusively to Sky and will talk to others on a confidential basis around her engagement scale, but given academic competition to publish there is only a limited amount she can say publicly.

 

Stuart McDonald (News International) and Euan Mackay (Kantar Media) – Show Me the Money: Proving the value of tablets

Given that the results of the research are being used to inform News International’s commercial strategy, they didn’t really go into how value was proved. The research was conducted among News International’s subscriber base, and tested interactive advertising on a beta app (The Times app doesn’t yet have advertising) against a premium engagement index, comprising of perceptions of an ad being

• Memorable
• Relevant
• Engaging
• Trustworthy
• Premium

 

Richard Curling (Google) – YouTube Skippable Pre-Rolls: Measuring the power of choice

Given “Hurry sickness” – the malaise where people feel short of time so perform tasks faster and get flustered by any delays – we’re increasingly looking for shortcuts.

YouTube “true view” means that users get to choose their adverts – if they don’t like an advert, they can skip it. Advertisers only pay for adverts that are viewed all the way through. Google interpret a high view rate as a high quality score, and this will factor in alongside price when bidding in an auction for advertising space. Thus, high quality ads are rewarded (though arguably very low quality advertising can benefit from a lot of free, interrupted views).

Using Ipsos MediaCT, Google tested the effectiveness of these ads using biometrics (heart rate, respiratory rate, skin conductance, motion- via Innerscope), depth interviews and eye-tracking. These found that both skipped and “true view” ads scores higher on their engagement metrics, though the true view ads scored highest. However, this wasn’t as clear-cut as you might expect – people opting in might have higher expectations and so could be harder to please. Conversely, the engagement of people forced to watch an ad might pick up towards the end as they get ready for their content to start

Richard’s recommendations for advertisers were to

• Entertain the user, since you are the content
• Be clear, and support user choice
• Embrace “natural” targeting

 

Afternoon sessions

I was paying less attention to these, since I was mentally rehearsing my speech

• Ross Williams and Becky from Ipsos MediaCT presented their “Big Brother Research – Who’s Watching Who?”, which combined social media monitoring of Big Brother properties. with Facebook polls. While Big Brother wasn’t as big as other properties, it had a 80-20 proportion of comments to likes on Facebook (indicating an engaged audiences), while alternative programmes had the opposite ratio
• Steve Cox of JC Decaux presented “Airport Live” – following a small number of passengers at both their departure and arrival airports to see what they were noticing
• Matthew Dodds of Nielsen and Nick Metcalfe of the Telegraph presented “Telegraph Print + Net Online Multiplier study” which took 5 groups of people (Telegraph print readers, Telegraph online readers, readers of both, non-print readers with matched demographics, non online readers with matched demographics) from UKOM to test uplift in advertising measures
• The Good The Bad & The Ugly of Media Research was hosted by Max Willey and featured myself, Dave Brennan, David Fletcher, John Fryer, Stef Hrycyszyn and Loraine Cordery talking about whatever we wanted to for three minutes. David Fletcher won the prize, for his tale of why people think they want online dashboards but don’t

 

Industry Updates

• BARB is looking into a non-linear database that would report on archive programmes on demand, and catch-up from longer than seven days after transmission. They will also evaluate, and possibly publish topline results of, the TV+online data
• POSTAR – now have tube and bus data, and are looking at GPS devices to see how people move around. This is being validated and they hope to get it into a reporting system soon
• NRS – concentrating on fusion with UKOM data, but hope to get more granular data and move online in future
• RAJAR – moving the diary online, and continuing to explore the viability of passive meters
• IPA – bedding down touchpoints. Touchpoints 3 included word of mouth, mobile internet, social media, gaming and on-demand. Touchpoints 4 will bring in tablets and apps, and change from a device-first structure to a content-first structure. It now has 60 subscribers (including each of the top 20 agencies) and has launched in the US. They are also piloting an app to go alongside the diaries
• UKOM – the past year has been about stabilisation after some data issues. The contract is currently out to tender and whomever is successful (they would take over in January 2013) would look to measure all devices and locations (ie beyond home/work fixed internet to include mobile and video)

sk

Advertisement

MRS Mobile Insights Conference

The first Mobile Insights Conference, hosted by the Market Research Society, took place in London this week. I was in attendance.

I’ve mentioned my thoughts on the word insight several times in the past. Did I take away many insights from the day? No. Were the talks useful? In parts. Was it worth attending? Yes

Although the first use of a mobile phone for research was over ten years ago, the techniques available on the devices are still in their infancy. This means we are still in the experimentation stage of understanding what can be achieved on mobile – compelling innovations and applications of research do exist, but are rare.

As such, the projects discussed over the course of the day were largely (but not exclusively) small and tactical. It would have been great to hear of any companies really embracing mobile research to inform strategy, but perhaps we haven’t yet got to this stage.

Given the relative lack of application, the day largely focused upon the mechanics of mobile research. A topic not particularly attractive to research buyers – I counted two research buyers on the delegate list alongside attendees from research agencies and suppliers (discounting speakers). While it important to collaborate as an industry, the realities of competitive advantage meant that, with the odd exception, research agencies aren’t going to be particularly candid about learnings when they are speaking to other research agencies.

Those gripes aside, I did take some several useful things from the day.

Tim Snaith, OnePoint Mobile Surveys – Defining the Future

After some opening remarks from this OnePoint colleague Neil Jessop, Tim Snaith took the floor to talk about the immediate future. Some of the points he made include:

• “If we define mobile research in a certain way, it will be adopted in a certain way”. Instead, we should be open-minded as how people ultimately choose to engage with the mobile will define how we can engage with them
• There is currently no knowledge base in mobile so we need to build foundations with case studies and open reports
• Downtime on the computer is decreasing because there is so much on there to keep us busy. Mobile is now the device for downtime, creating opportunities for surveys
• Blending computer and mobile research isn’t ideal, since the space restrictions on mobile mean you are effectively asking different questions
• International mobile research needs to consider the different regulatory and technological requirements in each market (for instance, different networks use different character sets in SMS)

Alex Wilde, Globalpark – Tracking trends

Alex defined the four types of research that can be conducted on a mobile as a survey (e.g. invited by a banner ad), diary, storytelling (capturing stimulus) and netnography.

An issue with surveys is that there are over 3,000 combinations of mobile handset, browser, operating system and so on. Ensuring usability is therefore a challenge. Within survey design, he advocated drop down boxes rather than radio buttons, and limiting text boxes to a single line field

With regards to some of their mobile studies

• They receive 35% of responses within the first hour (Lightspeed say they received 60% of completes in the first 15 minutes and 90% within the first hour – evidently research design will be more indicative than overall methodology trends)
• When survey respondents were asked after a few questions if they would like to continue the survey and, if so, where; 70% of iPhone users said they would continue on their phone but 90% of non iPhone users said they would prefer to continue on their computer
• 90% of respondents were willing to share their location via GPS

Tom Webber, Nielsen – Smartphone trends and opportunities

Tom shared several statistics with the audience:

• Smartphone penetration in the UK grew from 12% to 20% in Q1 2010 (smartphone defined by an operating system such as iOS or Android)
• As the prices of phones rises, long term ARPU falls (presumably this was down to unlimited data charges and the need to bundle SMS rather than charge individually?)
• Net Promoter Scores for smartphone owners are fifteen points higher than those for featurephone owners (26 vs 11)
• 64% of iPhone users use apps daily, compared to 52% of Android users and 38% of other phone users
• The Apple App Store has 84% satisfaction; the Android Marketplace has 81%. Apple performs better on range of apps and download experience; Android wins on ease of browsing and discovery
• American 13-17 year olds sent an average of 3142 texts a month in Q1 2010 (I believe this figure includes twitter and facebook updates); the figure for under 12s was 1152
• In the US, Twitter accounts for 42% of phone messages sent; Facebook for 16% (though I believe GSMA data has Facebook far ahead in the UK, at least in terms of time spent)

Patrick Hourihan, Yahoo! – APPetite

Patrick (Disclosure: an ex Essential employee although, like in Ghostbusters, our streams never crossed) was the first speaker to give a presentation, rather than read a report, and it was probably my favourite talk of the day in terms of interesting content (if not practical application).

After some general industry statistics (such as 11m social network users via mobile, 5.7m people downloading games and 11m using mobile media for entertainment, he ran through the methodology for APPetite

• Online diary and forum
• Focus groups in London and Manchester
• Depth interviews with in situ mobile usage
• 2,000 person online survey among 16-65 year old mobile media users (done online as mobile doesn’t give the depth or the same level of representativeness)
• Mobile app survey using Research Now’s real-time data collection tool (I think the questions were regarding which app was used, where and how it compared to online)

Things the research covered either qualitatively or quantitatively include

• Consumer expectations have shifted from functional use to emotional use
• Nearly as many use mobile for entertainment (58%, if you include social networking) as they do functional (63%)
• There is a claimed decline in snacking, with longer usage sessions
• Brands have been mapped to times of day, with the BBC, Facebook, Google and Yahoo! in use across the entire day
• Apps have the wow factor among users, and satisfaction of services (such as Facebook) is higher for the app than the for the web (NB: This wasn’t broken down by operating system)
• However, there is some confusion over what an app is (e.g. it could be a  shortcut to a website)
• 55% don’t have a preference between an app and web so long as they can fulfil their specific needs
• When a mobile service doesn’t work, 44% blame the brand, 34% the phone and 29% the network

Siamack Salari, Everydaylives – Using mobile for ethnography

Siamack’s talk was the only brazen sales pitch of the day, but it was justified since he was the only person speaking with a genuinely groundbreaking (or so it seemed to me) research technique – a mobile application where people can capture audio, video and pictures, tag it and upload it to a central server to be sorted, filtered, analysed and edited.They can comment and feedback on the information others compile.

The iPhone app cost £5,000 to develop (whereas the BlackBerry app cost around £60,000) and will allow ethnography to be conducted on a larger scale. Previously, he would film 2 hours of footage a day over several days, conduct interviews separately and then spend time editing the reams of footage.

The app come with a small price, but he is shortly going to launch a consumer version, which is free and would allow people to effectively sell their lives to willing buyers. He mentioned that, independent of this, Coca Cola and P&G approached him about fitting out their workforces with the application.

Learnings from his first four studies are

• Not all iPhone users are familiar with iTunes, and so will need to be walked through app installation
• A template of how to shoot video is required for a consistent quality of submissions
• Research needs to be designed carefully to avoid it altering people’s behaviour
• He has facilitated post-filming opt-outs, where he gives people an email address and a code, in case they want to have their privacy protected in their friends’ film

Leonie Hodge, Channel 4 and Anthony Cox, Sparkler – Using SMS to uncover drivers of viewing

As a cost effective alternative to ethnography (and allowing a greater sample), Channel 4 commissioned Sparkler to conduct research into drivers of viewing using 24 depth interviews, 9 in-home evening visits and 35 media diaries lasting for 6 days using a combination of mobile and paper.

Essentially, on days 1, 3 and 5 Sparkler would text the respondents to ask them what they were going to watch on TV that evening. They would send several messages (manually) if requiring clarification or reminding. On days 2, 4 and 6 people would fill in a standard consumption diary of what they watched the previous evening.

The research showed that there were very few appointment to view programmes (a disappointment?), and the majority of consumption was last-minute or spontaneous decision making.

Sparkler’s advise was to

• Get the stakeholders to pilot the technique, in order to engage them with the process
• Keep it as simple as possible for the respondent
• Design the outputs so analysis can be as efficient as possible

AJ Johnson, Ipsos MORI – Using mobile to gain greater customer insights

AJ was the exception in that he was quite open and candid about Ipsos MORI’s various experiments with mobile research – both successes and failures. Initiatives include

• Using mobile for passive media consumption for radio (via ambient sound recording) and posters (via GPS)
• Quick polls for PR purposes
• Comparisons to online research, which showed that mobile surveys took longer with shorter answers and a lower response rate, but that quality of response (recall, in this instance) was higher
• Multimedia diaries – they invited 1,000 people to take part in recording their weekend activities. 200 agreed to take part, but only 37 provided usable material. They had asked for 4 diaries a day from people, but received an average of 2.7 per person (they tended to be photos)
• Software they have been experimenting with includes Microsoft Pivot and The Link (unsurprisingly, I can’t find the link)

Gavin Sugden, T-Mobile – Measuring customer satisfaction via SMS

Gavin was also exceptional, in the sense that his was the only example of SMS research being used more strategically. T-Mobile changed their customer satisfaction survey from a 20 minute CATI survey to an SMS survey of around 6 questions – 3 standard questions and 3 from a battery of 15. They found SMS would be a better capture method and more cost effective than CATI, IVR or the mobile web, though Gavin didn’t seem to have a problem with privacy, saying that people could delete the messages if they didn’t want to respond (T-Mobile customers would be contacted within 24 hours of purchase in-store, web activity or phoning customer services).

With people only eligible to complete one survey in 60 days, they are seeing a 20% response rate.

They have an online reporting tool, which allows analysis of quantitative data and aut0-coding of verbatims.

They currently use the results for performance management (e.g. benchmarking stores) but are moving towards coaching and development of staff, and seeking of ways to improve customer satisfaction. They are also going to start include campaign awareness questions

Liam Corcoran, Fly Research – Insights into alcohol consumption

Liam walked us through a case study from a study he ran for Mintel, looking at alcohol consumption of young adults. Rather than hazy recollections, they used mobile research for in situ response. At 10pm each night, they would send an SMS with a link to a 7 question mobile internet survey.

He also talked a bit more generally about their mobile panel research.

• They tend to receive 60% response rates from their panels and 20% from cold sample
• 50% of their responses come within the hour and the vast majority within 24 hours
• He wouldn’t recommend a mobile internet survey of more than 15 questions
• Mobile is a great tool for a follow-up to online research, if just a couple of additional answers are needed

Sarah Sanderson and Jason Vir, Kantar Media – Integrating mobile with other research techniques

Sarah and Jason gave the last talk by going through a very recent project they ran during England’s World Cup campaign (presumably the project was ran specifically for the Conference; it must be nice to have the resources to be able to do that!)

The research was designed to access the emotional response in real-time, which they felt would be more honest and truthful

If I remember correctly, they:

• Recruited people to the project based on answers to questions in TGI Sport+ and TGI Postscript (the TGI omnibus)
• Got some people to take part in an online community to discuss the tournament
• Asked these people to submit videos and pictures of them during/after the game (I think 7 people did this, I’m not sure if the community was larger)
• Sent a WAP survey at half time to a larger sample. 207 responded, 90% of these during the half time interval

Neil Jessop closed the day with a few remarks, including a prediction that there would be more mobile surveys than online surveys. This was the one contentious comment of the day (in my opinion), and this sums up my overall feelings of the event. There were a few interesting asides and thoughts, but there was nothing at the event that really made me stop and think. A shame, but perhaps I was expecting too much.

sk

Image credit: Research-Live