Reproducibility: expect less of the scientific paper
Make science more reliable by placing the burden of replicability on the community, not on individual laboratories.
In 2018, we embarked on a journey to assess the reproducibility of biomedical research papers from Brazil. Thus began a multicentre collaboration of more than 60 laboratories to replicate 60 experiments from 2 decades of Brazilian publications1. We randomly selected experiments that used three common laboratory techniques: the MTT assay for cell viability, RT-PCR to measure specific messenger RNAs and the elevated plus maze to assess anxiety in rodents.
Each experiment will be repeated in three labs, and each lab has developed replication protocols based on the original article’s written methods. The process of building, reviewing and preregistering these protocols has taken months of communication between the coordinating team and the labs performing replications. We had intense arguments around the meaning of positive and negative controls and the merits of different metrics to define replication success. We also spent many hours on mundane tasks, such as studying the nutritional content of different brands of bologna sausage to better emulate a cafeteria diet fed to rats in one experiment.
These are just some of the obstacles we have faced so far as coordinators of the Brazilian Reproducibility Initiative: there was also the massive shutdown of labs due to the COVID-19 pandemic and the sharp decline of Brazil’s currency, the real. With all of this, experiments are starting slowly, and the project is now set to finish by the end of 2022.
That said, we have already reached conclusions that apply beyond Brazilian science. As a broad solution, more rigorous protocols and better descriptions of methods are important, but insufficient for reproducibility — and might not be feasible for every paper. Current requirements for wide-ranging experiments in a single article are part of the problem. To solve these issues, expectations placed on the scientific paper must change.
Reproducibility is costly
Research articles in the life sciences are more ambitious than ever. The amount of data in high-impact journals has doubled over 20 years2, and basic-science papers are increasingly expected to include evidence of how results will translate to clinical applications. An article in a journal such as Nature thus ends up representing many years of work by several people.
Still, that’s no guarantee of replicability. The Reproducibility Project: Cancer Biology has so far managed to replicate the main findings in only 5 of 17 highly cited articles3, and a replication of 21 social-sciences articles in Science and Nature had a success rate of between 57 and 67%4.
