Cennydd Bowles’s essay on “Datafication and ideological blindness” attacks the business/product-design vogue to base every decision on “data” on the grounds that data reflects objective truth, free from bias.
Datafication has many problems, but one leapt out for me: using A/B testing turns your customers into experimental subjects whom you try to nudge into a specific course of action, which excuses “unethical design” because “anything that moves the needle is fair game: no one is willing to argue with data.”
The alleged objectivity of data squeezes out vision and leadership — no one is responsible for deciding how to make the product better, that’s the data’s job — and products become incoherent reflections of statistical ghosts and contingent truths.
I’m a great believer that criticism need not offer solutions (I don’t have to know why your head hurts to observe that banging it against the wall isn’t helping), but it’s always better when criticism can also point to ways out, and Bowles concluded their essay with a set of recommendations for when and how to do statistically-driven product design and improvement.
If you’re putting data at the heart of your decision-making, you need to get it right. That means:
* employing skilled staff who will set up experiments accurately, avoid flaws such as p-hacking, and have the numeracy and statistical capability to draw valid insight from your raw data.
* investing in watertight analytics technology with excellent uptime and security.
* a laser-like focus on team efficiency and deployment. No point garnering insights if you can’t act.
* surprisingly large sample sizes. Thanks to the complexities of statistical power, you may find a minor tweak in a low-conversion process will need a >100,000-user sample for a valid test.
Without these, you may be making decisions off faulty data. Worse, you won’t even know. Thanks to the legitimising effect of datafication, you’ll feel highly confident while doing the wrong thing; betting on a hand of four ♠s and a ♣ that you misread as a flush.Data can of course be an enormously valuable strategic input, if these pitfalls are sidestepped. Senior designers and leaders can’t withdraw from the data discourse, but they are well placed to question its ideological power. Data is a valuable adviser but a tyrannical master, and in some companies datafication has such a stranglehold that other approaches are permanently in shadow.
Datafication and ideological blindness
[Cennydd Bowles]
(via 4 Short Links)