Ben Goldacre‘s Bad Science column frequently exposes less-than-stellar research findings that have been subjected upon the general populace. He’s always worth reading, and in the past has even inspired me to vent at some of the ridiculous claims.
But his latest column on PR reviewed data has, surprisingly, caused me to reconsider my stance.
In it, he compares the recent advertorials that the Express ran (which were outside of the rules) to these sorts of surveys. This led to a response from one of the main perpetrators – One Poll – and they (sort of) agreed.
In their own words, “We’ve been providing branded, stat-based news copy to the nationals for more than ten years now. Why do you think we do it? Everyone is aware this is a branding exercise…”
Unlike the advertorial row, the transaction is non-monetary and thus legal. The journalist is essentially outsourcing part of their responsibility. In some ways, it is like a landlord taking on a lodger, with the lodger earning their keep through “chores” rather than paying rent.
Does it matter? I don’t disagree with One Poll when they say one of these survey stories can be entertaining and provoke discussion – though I might replace the word “entertaining” with “momentarily diverting”.
But I disagree when they say that the surveys are valid. In terms of validity, I would subjectively rank this approach as thus (moving from invalid to valid):
- Made up data
- Straw poll of close friends
- Accumulating opinion from one source (e.g. comments from one news story)
- PR survey
- Hypothesis testing survey
- Exploratory survey
I have issues with their validity not so much because of the financial motivations of respondents, but because of an obvious inability to replicate the answers. They don’t have an objective truth.
To use one of their most recent press releases as an example: Man City fans are among the poorest in Britain. If the survey were repeated tomorrow, I wouldn’t be at all surprised to see a totally different result.
SIDENOTE: This post isn’t meant to disparage this sort of research (I’ve done that already), but for supposed experts they can surely create better copy than ”It’s no surprise to see Chelsea up at the top and the other big London clubs. They have a loyal ban base with supporters around the country and obviously have some money to spend.” Aside from spelling and grammar, perhaps they should have made the correlation between geographic disparity and affluence a bit more explicit…
While I still think this approach (to both research and marketing) is nonsense, I’m no longer against it appearing in newspapers. What I would like to see, however, is an explicit admission that it is a one-off accumulation of non-representative opinion at a single point in time (or something slightly catchier). It is not fact that Man City fans are among the poorest – it is just the result from a single, dodgy survey among people that live on the Money Saving Expert forums. These surveys are views, not news.
And with that caveat, I have no issue with this method. While labeled as market research, it is not something I, nor my company, would participate in and so we aren’t in competition with these practitioners. There is evidently a market for this sort of product, so good luck to those pursuing it.
One Poll have worked with some big names so they must be doing a good job in their niche. That niche is nicely summed up by their “No coverage no fee” results-driven model. The legal equivalents have been stigmatised as “ambulance chasers” – I wonder whether this type of service can avoid a similar slur…
Image credit: http://www.flickr.com/photos/daveknapik/
PS I’d be interested in hearing from anyone who has commissioned a PR survey of this nature. Outside of the value of the column inches being greater than the cost of the survey, has there been any tangible, noticeable benefit to engaging in this sort of activity?