Brian Christian's long Wired feature on A/B testing does a good job of explaining the quiet revolution in product design we've experienced this century, the modes of thought that habitual A/B testing encourages, and the drawbacks to those modes. A lot of the products and services we use today are designed to a turn that makes the previous technologies look like stone axes. That's largely thanks to the ability to run multivariate tests on vast sets of diverse design-choices and quickly converge on optimal solutions that are continuously and automatically refined.
For that same reason, A/B increasingly makes meetings irrelevant. Where editors at a news site, for example, might have sat around a table for 15 minutes trying to decide on the best phrasing for an important headline, they can simply run all the proposed headlines and let the testing decide. Consensus, even democracy, has been replaced by pluralism—resolved by data…
Google insiders, and A/B enthusiasts more generally, have a derisive term to describe a decisionmaking system that fails to put data at its heart: HiPPO—”highest-paid person’s opinion.” As Google analytics expert Avinash Kaushik declares, “Most websites suck because HiPPOs create them…”
One consequence of this data-driven revolution is that the whole attitude toward writing software, or even imagining it, becomes subtly constrained. A number of developers told me that A/B has probably reduced the number of big, dramatic changes to their products. They now think of wholesale revisions as simply too risky—instead, they want to break every idea up into smaller pieces, with each piece tested and then gradually, tentatively phased into the traffic.
The A/B Test: Inside the Technology That’s Changing the Rules of Business
