How to Spot Bullshit Science – 4 Common Mistakes

Do you ever wonder why there are so many contradictory claims in media about nutrition?Go on any big news site right now and you’ll find bold claims seemingly supported by scientific studies. The problem however, is that the vast majority of these headlines and articles are written by people who can’t be bothered to read or fully comprehend the study data.Whether you read blogs, newspapers, or watch TV shows for your information, you need to know how to spot poor journalism and science.There are many common mistakes made in regards to writing about studies, but in this article I’m going to go over 4 that I feel are the most frequent offenders.

1. Misinterpreting the Results

The main purpose of a journal publication is to present data, make some worthwhile observations, and conclude the main data findings.
Look at that menacing chocolate

Look at that menacing chocolate!

The problems and arguments begin when one group cites a study, let’s say for example a study on dark chocolate finds that it lowered total cholesterol levels, and proclaims “Chocolate is the best food EVER!”.Conversely, another person could see the exact same data, but see that cholesterol composition was unfavorable (less good cholesterol, more bad cholesterol), and say “Chocolate will KILL you!”.1 study, 2 different interpretations, and 2 hyperbolic claims. (By the way, I just made this up, this isn’t a real study).The data says what the data says, but is often taken out of context. Newspapers and magazines, and their online counterparts, often write headlines like these and then twist the study information to support whichever side they want.Why do they do this? Well let’s play a quick game of which Headline will attract more readers:

A) Dark Chocolate Lowers Total Cholesterol and Changes Cholesterol Composition, Implications Not Known


B) Breaking Scientific Study Finds That Chocolate Could Kill You

For the average reader, B wins hands down every time. People don’t notice the mundane, they are attracted to the extraordinary.Takeaway: The next time you read an article about some sort of nutritional breakthrough that seems hyperbolic, take a look at the actual study and see what the data really says.

2. Who Wants to Play the Telephone Game?

Who Wants to Play the Telephone Game

It’s for you!

The telephone game, for those who weren’t fortunate enough to play this at school as a child, was the game where everyone sits in a circle and a message is whispered from person to person from start to finish. You start with an ordinary phrase and end up with something completely unrelated.While it’s a fun game for kids, it becomes a lot less entertaining when you realize that this process is often repeated with scientific literature.A newspaper will make an inaccurate claim from a new study they’ve skimmed over, bloggers jump on the topic using the newspaper as a source and so on around the circle, or in this case the web.It’s frustrating to me that so many people use sources without doing due diligence on them. Bloggers in particular are terrible for this as a group, which is why the guys like Mark Sisson, who link to every single study they reference are enjoyable to read.Takeaway: Never take a claim at face value, always do your due diligence if it’s a topic that could significantly affect you.

3. Ad Hominem

An ad hominem is a type of fallacy, where an argument is based on the person delivering it.If you are reading an article about anything, nutrition included, your opinion on the topic at hand should not be swayed by the person writing or talking about it.I know everyone has experienced this first hand in one way or another. You’re talking with a friend and one of you says “Dark chocolate will kill you”, when pressed for a reason why, the response is typically along the lines of “It was on the news, it must be true”.Takeaway: Anyone trying to support or attack a statement based on the person delivering it should be pressed for factual information. If they can’t provide any you will have to look it up and decide for yourself.

4. Biases…Biases Everywhere

One important thing that university taught me was to never schedule something at 8 am on a Friday. Second to that it also taught me that even the smartest people have significant biases. It’s a human trait, and while can be managed, often finds a way to surface.The Scientists
Biases…Biases Everywhere

Your friendly neighborhood scientist

While scientists are often seen as brilliant, logic-driven, honest people, the scientists performing the studies are still human. Under-funded with immense pressure to get results in order to receive further funding, many scientists are willing to get ‘creative’ in order to achieve the results they desire.This can range from repeating a part of the study until it finally works, omitting data that doesn’t support their theory as outliers and even presenting data in a misleading way.Recently, a biotech firm called Amgen found that only 6 of 53 “landmark” cancer studies were repeatable. While some studies do get retracted when caught-out, not all are found.There have also been cases of outright fraud, which yes is illegal, but certainly not all researchers who have done so have been caught.Keep in mind that most scientists really do good work, just be aware that things like these happen from time to time.The MediaThis includes news corporations, bloggers, and anyone else who works based on the findings of the scientists.For most topics it is easy to ignore or discount studies that don’t support your theory or way of life and only trust the ones that do in someway, this is called confirmation bias. The best example for this is political, especially if you live in or follow the United States.There are some newspapers, TV channels, speakers, bloggers, who won’t say anything bad about their preferred political party. Over time if you hear good things non-stop in favor of one thing and bad things against another, your opinion will reflect that. This is why your selection of where you get your information from is important.In nutrition or fitness it’s the exact same thing. Carbohydrates for example are an all or nothing proposition for many people. There are of course uses for carbohydrates in the body, and also negatives of having too many.Do you really want to be getting nutrition information from somewhere that is closed-minded and can only see a topic in black and white? Personally I do not, but it’s up to you to decide for yourself.The Corporations how-to-spot-bullshit-science-4-common-mistakesAnd finally we get to the corporations. I’m not going to go on one of those "corporations are evil" rants, but I do think it’s a problem when they try to get involved in science.In less prestigious journals you will sometimes see a disclosed conflict of interest that says “X inc. funded this study”.Right away that’s a red flag, because the scientists conducting the study are motivated to support the corporation’s best interest to keep getting funding. This can show up in the results like we talked about earlier, or it can show up with a poor study set-up.Important Note: Just because it is a sponsored study does not mean that it should be dismissed. It’s just an indication that you should scrutinize the set-up and procedure. In prestigious peer-reviewed journals you don’t really have to worry about this as it’s already been done by other professionals in the field.Case StudyI’ll give you an example, this study was done looking at high fructose corn syrup. Right away if you check out the competing interests section you’ll see this:
'JM Rippe has received research funding from the Corn Refiners Association for the present study. The other study authors reported no competing interests."
Surely the Corn Refiners Association has no interest in trying to paint high fructose corn syrup in a positive light, Right?So here’s what happened, people were divided into 2 groups and fed either 10-20% of their total calories in the form of added table sugar (in one group) or high fructose corn syrup (in the other group). They were both given an equal amount of restricted calories, which was to examine weight loss. No real surprise, they found that the groups lost about the same amount of weight after 12 weeks.The study itself looks to have been run fine, but I take issue with the study set-up. How about we compare the health effects of high fructose corn syrup to something that isn’t terrible for you like table sugar is? Not surprisingly, it would be difficult to find funding for a study like this.The biggest problem occurs after, when all of a sudden there are articles rationalizing high fructose syrup consumption citing this study as an example.Takeaway: This was a long section, but what you need to take away from it is that while scientific studies are extremely important to expand the knowledge we have on any topic, there are multiple sources of biases throughout the study and reporting process that you should keep an eye out for.

Concluding Remarks

The point of this article wasn’t to have you dismiss any claim based on a study, but instead to demonstrate the ways in which science can be manipulated.You should maintain a healthy skepticism and look for multiple studies on any topic in order to get the most accurate overall picture before making conclusions.

Add Comment