Small lies, big lies, statistics. The (un)intentional misinterpretation of data can have massive effects on decisions, may that be about war, business, or just about any aspect of life.
Intentional misinterpretation does include omissions of key information. As I’m working with market research, I often face the problem of what should be told to the clients. There’s a genuine challenge where say base sizes produce unrobust data, which is a common problem with qualitative projects. My personal belief is that clients should be made aware of all information, good and bad. Managers often think differently about this.
We recently run a project for “a big social network site” looking into ad exposure on their site and its effect on online purchase, or even just generic (related) action. The outcome was that in the specific quant (not qual) group, there wasn’t a visible link between the two, and in the very few cases where link actually could be suspected, repeat rates were close to zero. Since however the client is a strategic partner and a big player of the market, the management decided to omit 90% of the findings and report back on those few cases only where link was suspected. No mentions of the repeat rate. Press releases were made, everyone happy. (Apart from me but at least my name didn’t go onto the slides. I wouldn’t have wanted it anyway).
We’re now working with a company that’s selling shoes. Our qual project involves looking at 10 (!!) people’s behaviour, examining how they research and buy shoes. Client is building a massive strategy on this. No comment on that one, but one of the recurring feedback from the panelists was that they didn’t consider that client’s products because they were/are boring. So we’re reporting back that we think the client needs better online and offline presence. Can’t tell them that the people on whom strategy will be based think that their products suck. Double facepalm.
What’s the point in agreeing into such projects in the first place with base sizes so small that the margin of error is epic high, and then not report back the key finding? We could as well spend the money on a few pints in the pub and make up numbers and stories…
ps. image credit: unknown, not mine, found it on facebook.