John Bohannon teamed up with a German documentary crew to undertake a crappy junk-science study on the effects of bitter chocolate on weight loss, and managed to push their hoax to major media outlets all over the world — here’s how.
First, they created a fake science institute, The Institute of Diet and Health, and then recruited a friendly MD to help them recruit a small number of volunteers for a weight-loss trial. The volunteers were split into three groups: a control, a low-carb group, and a low-carb group that ate bitter chocolate (“Bitter chocolate tastes bad, therefore it must be good for you”).
Once the three groups had pursued their proscribed regimens for three weeks, they ran the numbers. The low-carb group and the low-carb-and-chocolate group both lost about the same amount of weight, but then the researchers paged through 18 other factors they’d measured in the study — “weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc” — and cherry picked a couple of factors that looked better in the chocolate than in the low-carb group. On this basis, they were able to assert that adding chocolate to a low-carb diet made you lose weight 10 percent faster.
They wrote up a paper that contained obvious statistical canards — small sample size, bad sampling methodology, p-hacking, poor control group analysis — to an open access journal formerly owned by Biomedcentral, which charged them $600 for “peer review” and published it verbatim in their International Archives of Medicine, their “premier journal.”
Working with medical PR flacks, they created linkbaity press-releases and spammed them all around. The releases, combined with the credibility of the journal publication, was enough to get them headlines in media outlets from Huffington Post to The Times of India — it seemed that no one noticed that the “Institute of Diet and Health” was only a couple months old, that “Johannes Bohannon” was really John Bohannon, and that his PhD was in an unrelated scientific field.
The lesson isn’t just that journalists aren’t good at parsing out nutritional science news, but that the small number of rigorous findings from nutritional science are drowned out in the roaring sea of bullshit that can be trivially generated to fill our insatiable orthorexic need to figure out what we should be eating.
Here’s another use-case for keeping Statistics Done Wrong by your side at all times.
The only problem with the diet science beat is that it’s science. You have to know how to read a scientific paper—and actually bother to do it. For far too long, the people who cover this beat have treated it like gossip, echoing whatever they find in press releases. Hopefully our little experiment will make reporters and readers alike more skeptical.
If a study doesn’t even list how many people took part in it, or makes a bold diet claim that’s “statistically significant” but doesn’t say how big the effect size is, you should wonder why. But for the most part, we don’t. Which is a pity, because journalists are becoming the de facto peer review system. And when we fail, the world is awash in junk science.
There was one glint of hope in this tragicomedy. While the reporters just regurgitated our “findings,” many readers were thoughtful and skeptical. In the online comments, they posed questions that the reporters should have asked.
“Why are calories not counted on any of the individuals?” asked a reader on a bodybuilding forum. “The domain [for the Institute of Diet and Health web site] was registered at the beginning of March, and dozens of blogs and news magazines (see Google) spread this study without knowing what or who stands behind it,” said a reader beneath the story in Focus, one of Germany’s leading online magazines.
I Fooled Millions Into Thinking Chocolate Helps Weight Loss. Here’s How. [John Bohannon/IO9]
(via Metafilter)