• Follow Curiously Persistent on WordPress.com
  • About the blog

    This is the personal blog of Simon Kendrick and covers my interests in media, technology and popular culture. All opinions expressed are my own and may not be representative of past or present employers
  • Subscribe

  • Meta

Data should be used as evidence and not illustration

I read the Guardian article on journalist’s struggles with “data literacy” with interest. The piece concentrates on inaccurate reporting through a lack of understanding of numbers, and the context around them. “Honest mistakes”, of a sort.

Taken more cynically, it is an example of a fallacy that I see regularly in many different  disciplines (I’m loath to call it a trend as, for all I know, this could be a long-standing problem) – fitting data around a pre-constructed narrative, rather than deducing the main story from the available information.

This is dangerous. It reduces data to be nothing more than anecdotal support for our subjective viewpoints. While Steve Jobs may have had a skill for telling people what they really wanted, he is an exception rather than the rule. We as human beings are flawed, biased and incapable of objectivity.

Given the complexity of our surroundings, we will (probably) never fully understand how everything fits together – this article from Jonah Lehrer on the problems with the reductionist scientific method is fascinating. However, many of us can certainly act with more critical acumen that we currently do.

This is as incumbent on the audience as it is the communicator – as MG Siegler recently wrote in relation to his field of technology journalism, “most of what is written… is bullshit”, and readers should utilise more caution when taking news as given.

Whether it is due to time pressures, lack of skills, laziness, pressure to delivery a specific outcome of otherwise, we need to avoid this trap and – to the best of our abilities – let our conclusions or recommendations emerge from the available data, rather than simply use it to illustrate our subjective biases.

While I am a (now no more than an occasional) blogger, I am not a journalist and so I’ll limit my potential criticisms of that field. However, I am a researcher that has at various points worked closely with many other disciplines (some data-orientated, some editorial, some creative), and I see this fundamental problem reoccurring in a variety of contexts.

When collating evidence, the best means to ensure its veracity is to collect it yourself – in my situation, that would be to conduct primary research and to meet the various quality standards that would ensure a reliable methodology, and coherent conclusions

Primary research isn’t realistic in many cases, due to limited levels of time, money and skills. As such, we rely on collating existing data sources. This interpretation of secondary research is where I believe the problem of illustration above evidence is most likely to occur.

There are two stages that can help overcome this – critical evaluation of sources, and counterfactual hypotheses.

To critically evaluate data sources, I’ve created a CRAP sheet mnemonic that can help filter the unusable data from the trustworthy:

  • Communication – does the interpretation support the actual data upon scrutiny? For instance, people have been quick to cite Pinterest’s UK skew to male users as a real difference in culture between the UK and US, rather than entertain the notion that UK use is still constrained to the early adopting tech community, whereas US use is – marginally – more mature and has diffused outwards
  • Recency – when was the data created (and not when was it communicated)? For instance, I’d try to avoid quoting 2010 research into iPads since tablets are a nascent and fast-moving industry. Data into underlying human motivations is likely to have a longer shelf-life. This is why that despite the accolades and endorsements, I’m loath to cite this online word of mouth article because it is from 2004 – before both Twitter and Facebook
  • Audience – who is the data among? Would data among US C-suite executives be analogous to UK business owners? Also, some companies specialising in PR research have been notoriously bad at claiming a representative adult audience, when in reality they are usually a self-selecting sub-sample
  • Provenance – where did the data originally come from? In the same way as students are discouraged from citing Wikipedia, we should go to the original source of the data to discover where the data came from, and for what purpose. For instance, data from a lobby group re-affirming their position is unlikely to be the most reliable. It also helps us escape from the echo chamber, where myth can quickly become fact.

Counterfactual hypotheses are the equivalent of control experiments – could arguments or conclusions still be true with the absence of key variables? We should look for conflicting conclusions within our evidence, to see if they can be justified with the same level of certainty.  This method is fairly limited – since we are ultimately constrained by our own viewpoints. Nevertheless, it offers at least some challenge to our pre-existing notions of what is and what isn’t correct.

Data literacy is an important skill to have – not least because, as Neil Perkin has previously written about, it is only the first step on the DIKW hierarchy towards wisdom. While Sturgeon’s Law might apply to existing data, we need to be more robust in our methods, and critical in our judgements.  (I appreciate the irony of citing an anecdotal phenomenon)

It is a planner trope that presentations should contain selective quotes to inspire or frame an argument, and I’ve written in the past about how easily these can contradict one another. A framing device is one thing; a tenet of an argument is another. As such, it is imperative that we use data as evidence and not as illustration.

sk

Image credit: http://www.flickr.com/photos/etringita/854298772/

Frontline: The Merchants of Cool

Frontline: The Merchants of Cool is a fascinating, albeit highly cynical look, into the way teenagers and children are marketed to.

Narrated by Douglas Rushkoff, it is close to ten years old, having been first broadcast in February 2001.

One of the programme’s key themes is that teen culture is fast-moving and transitory. Yet hindsight has proved this to be false.

It would seem that the more things change, the more things stay the same.

The programme was made pre 9/11, pre X Factor, pre Youtube, pre Facebook, pre Obama and pre Spotify (to name just half a dozen things that have shaped our entertainment culture in the intervening years). Yet it remains highly recognisable and relevant to teen culture today.

And so despite the assertion being wrong, it still remains required viewing for marketers, researchers and media folk – each of whom have the spotlight placed on them within the programme.

Discovering what teens want

Even back at the turn of the millennium, teens were seeing 3,000 discrete ad impressions per day, meaning that they would have been exposed to 10m of them by the age of 18.

Yet the programme asserts that surly teens are unresponsive to brands – they instead respond to what they perceive to be cool.

In order for content makers and marketers to know what kids think is cool, they need either formal research or an informal direct line to teens. The programme highlighted four methodologies used:

1. Cool Hunting – as typified by Dee Dee Gordon and Sharon Lee’s’s Look-Look. They start of by identifing teen influencers – early adopters, vocal advocates and people that regularly explore outside of their regular sphere of interest. After speaking to these people to find what they think is cool, they might recruit and train these kids up to be correspondents. They in turn go off and identify the next generation.  All information goes into a database that their clients pay a subscription fee to access.

This seems like a great business model for several reasons

  • It is a relatively low-cost model of both intelligence gathering and recruitment, meaning that the company can operate on a relatively small income
  • Due to the proliferation of research agencies and consultants, they will never have more than a small share of the market and so anyone that subscribes to the database stands a good chance of having a comparative advantage over rivals sourcing their information from elsewhere
  • The business is perpetual. Once something is identified as cool, it has been taken away from the cool kids and so is no longer cool. Thus they need to move onto the next thing

2. Under the radar marketing – Represented by Cornerstone Promotions, this odious tactic pays kids to “smuggle messages” onto forums or even in conversations, essentially paying kids to be walking, talking billboards without disclosure.

3. Ethnographic visits – Self-explanatory (though perhaps it was less well-known in 2001), where researchers and execs go spend time in people’s homes to observe them in their natural environment

4. Screen tests – Inviting kids to test to agents for various entertainment professions. Jessica Biel was discovered in one of these tests.

From these techniques, two key role models/personality segments were discovered – one for males to aspire to and one for females

  • The Mook – where arrested adolescence and crudeness are celebrated, typified by Tom Green and Jackass
  • The Midriff – where your body is your best asset so flaunt it even if you don’t understand it. Britney and Christina were the key role models

Does it sound familiar? So does teen culture perpetuate across generations, or are we on an irreversible trend towards sexualised stupidity?

A critique of these research techniques – and research in general – is that they don’t understand teens as people. Instead they are just customers. After all, the industry is named marketing research and not human research.

Although this is primarily a semantic argument, I think that, broadly speaking, the programme makes a valid point and it is still something that hasn’t been properly addressed in either marketing research or brand/market planning. Something to think on.

Content and marketing trends

Without a true understanding, it means there is essentially a giant feedback loop in play. The media sells kids images of themselves to themselves, and they in turn aspire to it.

There was the example of a Sprite party on MTV. Guests were paid $50 to show up, artists that played got paid and PR, MTV got cheap, aspirational TV and record labels got their exposure and sale. Yet, while it seems to benefit everyone, it ended up being quite conspicuous marketing,  and thus a turn-off for teens.

To be new and exciting to teens, boundaries need to be broken. After all, teens are about rebellion and anti-authority. If Dawson’s Creek is primetime (don’t laugh, it did cover some pretty edgy themes at the time), specialist outlets need to up the stakes. This meant that counter-culture icons such as the Insane Clown Posse and Limp Bizkit eventually got packaged up and sold to teens

A line from the programme I liked is that ICP are “so crude and intolerable that they are essentially indigestible”.

So is this an irreversible trend to the gutter? The edges are always different to the centre, and it would appear that for anything to reach a teen mainstream it needs to be largely digestible. Despite the questionable authenticity, contemporary trends such as 3oh!3 and Look at this fucking hipster seem largely harmless, while  Jersey Shore et al keep up the trend of sexualised stupidity.

Frontline have maintained an excellent website with a full discussion of the trends and coverage of the various interviews – it can be found here. The programme is also available to watch in full – either there or on Youtube (part 1 is embedded below, for RSS readers)

The situation today

The programme does maintain a sense of middle class adult bemusement throughout, but it still makes some great points about the attempts of marketers to pass off fake authenticity, with continually more explosive and extreme angles that try to stand out in the onslaught.

Yet one thing that has exploded over the past decade is the sheer supply of media and entertainment – I’m sure these 3,000 impressions a day have been far surpassed (particularly with the rise of branded entertainment). So marketers are needing to move away from disruption to find ways of actually engaging with teens on their terms – through opt-in involvement rather than unwanted interruption.

Will this fragmentation reduce the power of the mainstream? Rather than two teen idols, is there now two dozen? I’m not so sure. Teens have always been tribal and the hierarchies of social groups within schools aren’t going to fundamentally shift as we get more technologically advanced and savvy. They may not sell as many tickets or items as they would have a decade or two ago, but Lady GaGa, UFC and the Twilight saga are just as important cultural icons for teens today as Madonna, WWE and Point Horror were for previous generations.

But rather than focus on the markets, should we shift our focus back to the audience? I think so. Researchers and planners should essentially be an ombudsman for their target markets – representing them with a clear voice to be heard and respected when designing strategies and tactics.

sk

Reblog this post [with Zemanta]