The Replication Crisis
Science is a little less true than we thought
“The good thing about science is that it’s true whether or not you believe in it.” -Neil Degrasse Tyson
“Ninety-seven percent of original studies had significant results [while only] thirty-six percent of replications had significant results.” -a metastudy on the poor replicability of literature in Psychology
Science is in the midst of a crisis, a crisis of replication. Multiple large-scale metastudies have attempted to replicate results from fields ranging from psychology to medicine and each time a good portion of the original results turn out to be wrong or overblown. The metastudy quoted above showed that, out of a hundred psychology papers, less than half were successfully replicated. Another metastudy found that only 13 out of 21 social and behavioral science papers could be successfully replicated. Another showed that out of forty-nine medical papers, 32% were wrong or overstated. As members of a modern liberal society whose very bedrock is supposed to be the unassailable truth of science, we ought to be alarmed by these results.
I recognize that questioning the validity of science is dangerous business. Climate change and the wider ecological crisis demand immediate action while doubt about them decreases our species’ chance of survival. Additionally, if significant people stop using vaccines then we may face the return of terrible diseases. From medical breakthroughs, to the incredible technology that surrounds us, the effectiveness of science has been proven time and time again. Science is clearly true, but it’s just not quite as true as we thought.
My personal experience as a PhD student in physics confirms on a small scale the larger replication crisis. Here are just a few things I witnessed personally:
When one experiment gave an exciting result, while an identical experiment did not, a paper was written that only mentioned the first experiment.
After being questioned on one of his assumptions, a scientist responded “Well we have to make assumptions,” and carried forward without missing a beat.
After a graduate student said to a professor “If we don’t make hypotheses before we conduct experiments, then are we really doing science?” the room got tense and the professor changed the subject.
A colleague mentioned to me that it seemed like professors could always find some way to “justify” a claim, even if the claim were not exactly true.
After being told by his student that the error bars on a set of plots the student made did not account for all the error, the professor responded, “These error bars will just be underestimates then.” The error bars were never revised.
Why is there a replication crisis? What incentives are there to write half-true scientific papers that sound exciting but misrepresent the truth? Well, what does any scientist want but financial security and the respect of their peers? i.e. positions at elite universities, awards, tenure, grants, and publications in prestigious journals. The desire for these things is matched only by their scarcity. The incentive for you, the aspiring scientist, is to stretch your definition of honest scientific practice to its absolute limit in order to make your research sound as exciting and original as possible so you can get that Nature publication which you will need in order to get an NIH grant which, you know, will be necessary to pay for lab equipment which will be needed to publish more exciting and original papers, without which there is no chance of impressing that stickler of a department head whose approval you need to get tenure, and, oh god, what if you don’t get tenure, will some other university hire you or will you have to abandon your dream and end up writing code for some software firm and what would people think of you then?
Equally alarming to the replication crisis itself is how this crisis is not being taken seriously by scientists. In eleven years of formal scientific education, and after having worked in five separate labs, not once did any of my professors or colleagues mention it to me. All that I have learned about it I had to research for myself. And the small percentage of scientists who do write about it tend to do so with a strange positive spin. Here is one article titled “The replication crisis is good for science.” Or, from the study quoted at the top of this article: “If initial ideas were always correct, then there would hardly be a reason to conduct research in the first place,” as if a replication crisis every now and then is a healthy part of the scientific process rather than an indicator of perverse incentives rooted deeply in the institution.
What we need is a system that incentivizes prudent & honest science rather than overhyped science, a system where you can write a paper that says “We know this experiment is unoriginal, but we thought it would be prudent to try and replicate another group’s experiment,” or “Here is an experiment that did not work.” Papers like these would contain information of enormous value to other scientists, but would be almost impossible to publish in today’s scientific climate. We need a system where you are celebrated for working on something similar to another scientist, rather than punished for being “scooped,” the term for when someone publishes a paper before you on your topic of research. The solution may be to evolve science beyond the current model wherein scientists submit papers to journals who then accept or reject those papers, a model that is clearly a relic of the age of print. The Internet offers infinite possibilities for dissemination of scientific results if we are courageous enough to try them. Promising options include Researchgate, a sort of social media for science where scientists can ask questions and publish their hypotheses and results in real time, SciNote, an electronic lab notebook that can be shared, and, of course, the glorious pirate website Sci-hub where just about any scientific paper can be downloaded immediately for free.
So what do we do in the meantime before science is revolutionized into the robust, honest institution that it ought to be? Should we treat science as real or fake? Well, obviously it is real: scientific discovery is what makes possible all the incredible technology that surrounds us. It’s just that it is not quite as real as we were led to believe. Climate change is a huge, very real problem and some of the models are probably wrong. Vaccines are a powerful, effective medicine and there is overwhelming financial incentive from the pharmaceutical industry to overstate their necessity. This space in between the two camps of thought is uncomfortable to inhabit, but a commitment to truth compels us to do so.
Part of this discomfort is that if we cannot depend wholly on science for truth, we must get some of our truth from other places. And here we actually do get to a positive side of facing the replication crisis which is that with this new uncertainty we are called to consult sources of truth that have been neglected for too long.
One of these sources of truth is personal intuition. Have you ever read a study and felt in your gut that something was wrong with the result? Don’t sacrifice that gut instinct. There is an attitude prevalent in society today, as demonstrated by the Neil Degrasse Tyson quote at the top of the article, which states that your inner knowing is inferior to the knowledge derived from science. In reality, both are valid sources of truth and must be measured against each other. As an example, I went into my first acupuncture session with a tight neck and walked out with more neck flexibility than I could remember ever experiencing. How can it be that acupuncture has been an effective treatment for me and many people in my personal network, but has not yet been proven effective by Western science? I do not know, but in this case I trust my personal experience over science.
The other source is indigenous wisdom. We in modern liberal society, in recognition of the massive damage inflicted on indigenous people, want to respect their teachings, but somehow cannot seem to treat these teachings as anything more than quaint reminders to be environmentally friendly before getting to the real facts about CO2 levels and emissions targets. The truth is, though, that many indigenous communities have been living in the same place for thousands of years and so are the only people on the planet who actually know how to live sustainably. There is real wisdom to be learned by relaxing our certainty in the rationalist materialist model of the world and trying to feel the truth of indigenous peoples’ models which tend to be more animistic. So, when an indigenous person, like shaman Martín Prechtel for example, says about climate change “that all the world’s weather wildness is the grief-binge of the Earth herself,” consider that it may be literal truth and not just metaphor.