Skip to content
“When it comes to chocolate, resistance is futile." - Regina Brett, best-selling author
As many readers know, I’ve expressed skepticism towards health study findings that suggest that a certain behavior causes rather than correlates with a health result. As I wrote in a blog two months ago, “these studies make great headlines, but may lead us to change behaviors without actually impacting our health.”
Therefore, a recent study on the benefits of chocolate consumption to weight loss caused me to raise an eyebrow. Yes, that’s right: a scientific study published last month claims that eating more bittersweet chocolate results in faster weight loss.
Hold on. What? My skepticism kicked in when I heard about this. I reviewed my internal checklist: Was there a control group? Yes. Okay, but were groups evaluated randomly selected? Yes. Hmmm… Fine, but were the results statistically significant? Yes. The “p-value,” the chance of the result being just a random fluctuation was less than 0.05, the standard for statistical significance in scientific studies. In fact, the average chocolate-eating participants lost weight 10 percent faster than the control group.
So, it’s true: eating chocolate is the sweet secret many of us chocoholics have been dreaming about all of our lives! Articles published in The Huffington Post, Shape Magazine, and Prevention Magazine, among others, touted the findings. Shape Magazine’s article was titled “Why You Must Eat Chocolate Daily.” It turns out that my skepticism was unfounded. My hesitation to believe health studies prevented me from accepting new, correct information. I felt like the relenting laggards who finally concluded that Galileo was correct about the earth revolving around the sun. Last to the party again. And it was my fault, as usual. And this time, I missed out on weeks of chocolate binging. Sad. My sadness could only be cured, of course, by chocolate, which in turn would help me get my mankini body according to the study, and all would end well, right?
Then the bombshell dropped. It turns out that although the data from the study was technically correct, the findings were completely worthless. In fact, it was a prank. A prank that didn’t require any lies, but just the manipulation of facts, with the help of logic, to show how easy it is for us to believe statements about the benefits of things like chocolate to our health.
John Bohannon is a German journalist who led the research, which was really an experiment on our naiveté. His team recruited participants for the study from Facebook, tested them to ensure they didn’t have any dietary sensitivities, and randomized them into three groups: one followed a low-carb diet, another followed the same diet and added a one-and-a-half ounce bar of dark chocolate, and the final group served as the control group and didn’t change their diet. The three groups were weighed daily for 21 days and concluded the study with a thorough round of blood tests and a written questionnaire. 18 different measurements were evaluated, including weight, quality of sleep, and cholesterol levels.
The two diet groups each lost five pounds after the three-week period, while the control group remained the same on average. Compared to the low-carb only group, the chocolate eaters lost weight 10 percent faster, had lower cholesterol levels, and higher reported “well-being” scores from the questionnaire. Each of these differences was statistically significant.
So, what was the prank? It was a legitimate study, right? Well, for the same reason why it’s so easy for brokerages to cherry-pick a “hot stock” from a list of 100 stocks and claim they’re great at beating the market, the results were garbage. The reason is that there were only 15 total participants in the study, and with 18 measurements evaluated, there was a good chance that something would turn out to be statistically significant. As Bohannon explained, “Think of the measurements as lottery tickets. Each one has a small chance of paying off in the form of a ‘significant’ result that we can spin a story around and sell to the media. The more tickets you buy, the more likely you are to win. We didn’t know exactly what would pan out—the headline could have been that chocolate improves sleep or lowers blood pressure—but we knew our chances of getting at least one ‘statistically significant’ result were pretty good.” Bohannon’s team had a study with an intriguing finding, and then hit the accelerator on the public relations machine. They even paid freelancers to write an acoustic ballad and a rap about chocolate and weight loss to boost the excitement.
As I mentioned earlier, this study was picked up by major publications. None of them, according to Bohannon, reported that there were only 15 participants in the study. Bohannon said that this is not surprising and “for far too long, the people who cover this beat have treated it like gossip, echoing whatever they find in press releases.”
It turns out that it’s not too hard to conduct and publish a weight-loss study (Bohannon did it in about one month with limited resources). However, it’s extremely difficult to conduct and publish a valuable study. Dr. Peter Attia, a surgeon who cofounded Nutrition Science Initiative, a nonprofit, contends that even well-funded, serious weight-loss studies often result in inconclusive findings. The Women’s Health Initiative, for example, “spent $1 billion [on a diet and health study] and couldn’t even prove that a low-fat diet is better or worse,” says Dr. Attia.
So, the next time I read a new study about a surprising new cure for excess gas or special ingredient to make me taller, I’ll rely more on my skepticism, even if I read an article in a mainstream website or newspaper.
We'd love to hear from you. How do you feel about this prank? Comment below or on our Facebook page at facebook.com/flytefitness, or tweet us at @flytefitness.
Be Flyte Fit,
Jeremy Greenberg
Co-Founder & CEO
Flyte Fitness
P.S. Don’t forget to sign up for health & fitness updates by clicking the box at the top of this page!
Continue Reading