The gold standard for researching the effects of diet on health is the self-reported food-diary, which is prone to lots of error, underreporting of “bad” food, and changes in diet that result from simply keeping track of what you’re eating. The standard tool for correcting these errors comparisons with more self-reported tests.
As if that wasn’t bad enough, eating correlates with everything, so if you go “p-hacking” through the data, looking for correlations after the fact.
My take on all this is that if there was a gross, easily observable effect from eating food that humans have been eating for hundreds or thousands of years, we’d already know about it. That’s why the amazing study showing that kiwi fruit promote good sleep turns out to have only 24 subjects, only two of them men, to rely on self-reporting, and to be funded by the kiwi industry. That’s why the incredible news that tomatoes improve memory turns out to be a preliminary result of a small study on very old Japanese people. If you’re not very old and of Japanese descent, that finding means something between nothing and so little as to be indistinguishable from nothing.
So a good rule of thumb, as far as I’m concerned, is that any nutrition study that finds any genuinely surprising, large-scale effect, is bullshit. Tiny, difficult-to-disentangle effects? Sure, there’s probably a lot of those. But “Eating peanuts makes you grow extra nipples on your ass-cheeks?” If that were the case, we’d have noticed by now.
Although concerns about self-reported dietary intakes have been around for decades, the debate has come to a head in recent years, said David Allison, director of the University of Alabama’s Nutrition Obesity Research Center in Birmingham. Allison was an author of a 2014 expert report from the Energy Balance Measurement Working Group that called it “unacceptable” to use “decidedly inaccurate” methods of measurement to set health care policies, research and clinical practice. “In this case,” the researchers wrote, “the adage ‘something is better than nothing’ must be changed to ‘something is worse than nothing.’”
The problems with food questionnaires go even deeper. They aren’t just unreliable, they also produce huge data sets with many, many variables. The resulting cornucopia of possible variable combinations makes it easy to p-hack your way to sexy (and false) results, as we learned when we invited readers to take an FFQ and answer a few other questions about themselves. We ended up with 54 complete responses and then looked for associations — much as researchers look for links between foods and dreaded diseases. It was silly easy to find them.
You Can’t Trust What You Read About Nutrition
[Christie Aschwanden/538]
(via 3 Quarks Daily)
(Image: Brassica oleracea, an edible wild plant, MPF, CC-BY)
