Replicability in the context of systematic reviews: A call for a framework with considerations regarding duplication, overlap, and intentionality
Letter to the editor
Reproducibility and replicability are considered central to the development and evolution of science, as they should ensure the efficient generation of reliable knowledge [[1. Much has been written about the “reproducibility crisis” [[2 and “replicability crisis” [[3] as a serious threat to all experimental sciences.
However, it has been noted that there are inconsistent and contradictory uses of the terms reproducibility and replicability by different disciplines, institutions, and groups. Other similar terms, such as repeatability, have been used to denote the general concept of confirming the research results of other researchers. Thus, the lack of standard definitions hinders the assessment of reproducibility, and applicability [[1]
]Although there are manuscripts pondering the definitions of reproducibility and replicability in general [[4] we consider that there are particular considerations regarding these definitions in the context of systematic reviews that deserve further attention.
]Although there are manuscripts pondering the definitions of reproducibility and replicability in general [[4] we consider that there are particular considerations regarding these definitions in the context of systematic reviews that deserve further attention.
When it comes to systematic reviews, multiple articles offered a definition of terms reproducibility, and replicability. For example, Shokraneh [5)used the following definition [quote]: “reproducibility is re-conducting the same study, using the same methods and data by a different researcher or team and the replicability is re-doing the same study to gather new data or recollect the data” [[6]
]Others reported that reproducibility involves reanalyzing the data collected in a study using the same computational steps and analytic code as the original study, and replicability is conducting a new study designed to address the same question(s) of a prior study [[7]
]Others reported that reproducibility involves reanalyzing the data collected in a study using the same computational steps and analytic code as the original study, and replicability is conducting a new study designed to address the same question(s) of a prior study [[7]
].The term reproducibility appears to be used consistently in the context of systematic reviews, basically meaning the use of the same data, and analytic methods as the original study (synonymous with computational reproducibility) to try to reproduce the original results.
However, the definition of the term replicability in the context of systematic reviews is challenging. “Addressing the same research question” and “re-doing the prior study” may be done with different methods, and in that case, it is difficult to expect true replication as the impact of different methods will hardly be assessable. The same research question can be addressed in various ways, introducing the notion of “replicability with a twist.”
In the manuscript describing the consensus checklist for replicating systematic reviews of interventions, Tugwell et al. acknowledged that terminology, and conceptual framework for replication were not standardized. They defined two types of replication of systematic reviews of interventions as either direct or conceptual. Direct replication was defined as a “purposeful repetition to verify the findings of the original research question,“ and conceptual replication as, purposeful broadening or narrowing of the research question in existing systematic reviews (eg, across broader or more focused populations, intervention types, settings, outcomes, or study designs)” [[8] .True replication of a prior systematic review would replicate precisely the same methods and see whether the same results are achieved. Replication is, basically, a duplication [[9]. That would be in line with the definition of “direct replication” [[8]. This should be feasible if the methods of a prior systematic review were fully and transparently reported, i.e., if a systematic review has the necessary attributes of “replicability” [[10]. However, when authors investigate the same research question with modified methods or modify the research question, it is not realistic to expect true replication. The question is whether the “conceptual replication” should be considered replication at all. In that case, it is realistic to expect to produce overlapping systematic reviews but not replicated/duplicate systematic reviews.
When analyzing the literature across different disciplines, Vachon et al. adopted a broad perspective of replication, including the repetition of an index study, the extension of an index study, and the road testing of a theory [[4]
]. However, just like suggested definitions of direct, and conceptual replication of systematic reviews [[8] this implies intentionally replicating a study.
]. However, just like suggested definitions of direct, and conceptual replication of systematic reviews [[8] this implies intentionally replicating a study.
An important aspect in all these definition considerations regarding the systematic review replication is the aspect of intentionality. A group of authors may intend to replicate an existing systematic review and plan it a priori as either a direct or a conceptual replication. However, we believe that it can be safely assumed that the vast majority of overlapping systematic reviews are not intentionally replicating previous reviews. The authors may not be aware of existing systematic reviews because they may not have searched literature adequately or analyzed registries such as PROSPERO [[11]
. In that case, they are not doing a purposeful replication in any sense. Such “non-intentional replication” reviews will then be considered a replication simply because they have a very similar/same research question compared to the existing review(s). In all non-intentional replications of systematic reviews, this brings up the consideration of what we mean by the same/similar question. Even if the clinical question is really exactly the same, there can be differences arising from the PICO and in the next step from the eligibility criteria. Language restrictions, for example, could play a large role here [12]. Reproducibility and replicability are expected to reduce research waste and increase the value of evidence [[13]. Although excessive duplication of systematic reviews [9] is certainly contributing to research waste, there may be legitimate reasons for conducting new, overlapping systematic reviews. But, then, legitimate overlapping systematic reviews should not necessarily be considered replicated, unnecessary, and contributing to research waste.
. In that case, they are not doing a purposeful replication in any sense. Such “non-intentional replication” reviews will then be considered a replication simply because they have a very similar/same research question compared to the existing review(s). In all non-intentional replications of systematic reviews, this brings up the consideration of what we mean by the same/similar question. Even if the clinical question is really exactly the same, there can be differences arising from the PICO and in the next step from the eligibility criteria. Language restrictions, for example, could play a large role here [12]. Reproducibility and replicability are expected to reduce research waste and increase the value of evidence [[13]. Although excessive duplication of systematic reviews [9] is certainly contributing to research waste, there may be legitimate reasons for conducting new, overlapping systematic reviews. But, then, legitimate overlapping systematic reviews should not necessarily be considered replicated, unnecessary, and contributing to research waste.
Thus, we should tread carefully when defining replicability in systematic reviews. Due to the nature of systematic reviews and their increased production, the context of replicability in systematic reviews is different than in other studies. The definitions themselves could lead to further confusion and unnecessary labels of research waste. Considering the concepts of duplication, overlap and intentionality, we consider that there is an urgent need for reconsidering definitions, and the development of a coherent framework of replicability in the context of SRs.