I talk a lot about the importance of context in understanding science. The results of one, single research paper do not tell you everything you need to know on a given subject. Instead, you have to look at how those results fit into the big picture. How do they compare to the results of other studies on the same subject? Have the results been independently verified? How do the specific experiments being done influence what you can and cannot say about the results? What questions aren’t answered by the study, and what new questions does it bring up?
You should be thinking about that every time you see anybody talk about the results of a single, new study. Without context, you get situations like this one, described by Travis Saunders on the Obesity Panacea blog:
Earlier this year my friend and colleague Valerie Carson published an interesting paper examining the health impact of various types of sedentary behaviour in a sample of 2500 children and adolescents. They created a clustered risk score (CRS) which took into account a child’s waist circumference, blood pressure, cholesterol, and inflammation, and then examined whether it was associated with 3 different measures of sedentary behaviour – accelerometry (an objective measure of movement), self-reported TV watching, and self-reported computer use.
Here is what they found (emphasis mine): For types of sedentary behavior, high TV use, but not high computer use, was a predictor of high CRS after adjustment for MVPA and other confounders. Here is what the Daily Mail had to say: Watching TV most damaging pastime for inactive children, increasing risk of heart disease.
Last month, our group in Ottawa published another paper (led by Dr Gary Goldfield) looking at different types of sedentary behaviour and heart disease risk factors in a cohort of overweight and obese teens (in contrast, the earlier study was on a sample of nationally representative youth). Interestingly, we found that neither TV time nor computer time was associated with increased risk in this group – in our dataset it was video games that were by far the most important sedentary behaviour.
Why is this a problem? Put yourself in the shoes of someone who just read the Daily Mail article, and who now believes that TV viewing is the single most damaging sedentary behaviour for kids to engage in. What reaction are you going to have when you read a similar article about our new study, suggesting that TV viewing and computer use aren’t important at all, but that video games are actually “the most damaging activity an inactive child can indulge in”?
As the source of this problem, Saunders rightly calls out journalists for pushing every individual study as a “GROUNDBREAKING NEW FINDING”. It is, unfortunately, rare to find TV and newspaper coverage that treats new studies in context, rather than as the final word. But to that, I’d add university PR people. The sad truth is, with newspaper layoffs, many of the people writing about science aren’t specialists. They cover city council one day, school board the next, and a new research finding after that. The press releases they get (and I know, because I get those press releases, too) push GROUNDBREAKING NEW FINDINGS not research that fits into a larger context. It’s the journalists job to know better. But it’s also the university’s job to not manipulate journalists.