Identifying Good Science
Most people probably encounter at least one article or social media post a day that starts with something along the lines of "studies show..." or "recent research suggests..." to entice them into some conclusion about a range of topics. Most of us understand how clickbait works (headlines that are much more dramatic than the actual article, but we fall for them anyway because it's easier to read a headline than a full article. But before you jump to conclusions, it helps to understand a few aspects of good and well explained research.
If these aspects aren't mentioned in the article, it may have just been the fault of the author of the article rather than the researcher but it means that you may not be getting what directly happened during the study or what it means for the topic as a whole.
- Was it an experiment?: It's very easy to observe correlations and draw out a conclusion. Observational studies are good for getting a feel of the research topic or establishing hypotheses for further research but are not as strong as experimental conditions where the researcher took extra care to determine specific causes and effects.
- Who funded the research?: This question has come up during many of my applied courses as a cognitive scientist and a neuroimaging student. Certain organisations may have the resources for research but are not basing their research on a testable or falsifiable hypothesis. This would be the difference between asking if there was an effect versus what effect when no previous research or observations indicate a relationship between what you are testing. There are people who refuse to do work that is too privately or commercially funded because there can be politics and cherry-picking of results to prioritise a non-scientific agenda.
- Is it a representative sample?: There's a statistical side to this question that asks whether the sample size is too big or small. To small means that the individuals or entities in the study unfairly represent bigger portion of the real population than they should. Too big runs into the problem of finding certain outliers more significant than they should be. A study may statistically have the right number of people but not be representative in other ways. This can commonly happen if a study is based on first year undergraduates rather than taking the time to use a more appropriate selection regarding age, gender, socio-economic background, religion or whatever else makes sense.
- What is the scope of the conclusion?: It is really important to understand the scope of the results from research. This includes the specific insight as well as limitations. Anyone who has written an academic paper knows that a large portion is spent on discussing results, assessing the initial hypothesis or question, taking into account limitations in the data or study, and what the next question or further research might be. This allows the reader to see the bigger picture if they are not an expert in the particular field.
The Take Home Message
If there's anything you should take from this post, it's that there is no ideal number of samples or type of research you have to look for when reading about research. You want to look for confidence that the researcher had a clear question in mind and did everything in their power to fairly and objectively measure the outcome. It is okay for people to explain observational data as long as they explain that they were observing and not testing a cause and effect.
Thanks for reading. I hope these posts and infographics are helpful. If you have any questions or comments, feel free to post below or email. And don’t forget to follow on Facebook and Twitter for updates and additional interesting articles.
Lastly, if you like what you read and want to keep up-to-date on the Daotive brand, subscribe to the mailing list now!