UIUC Quant Brownbag

Replication results are becoming a default criterion in assessing the credibility of scientific results. We have concerns regarding this practice and argue that most experiments are not replication-ready. A careful reading and analysis of research before rushing to replicate or taking replication results for granted is necessary for sound scientific practice. A replication experiment that does not carefully reflect on the design of a target experiment is likely to repeat its mistakes and inherit its inferential limitations. Only experimental designs employing valid methods that yield unambiguous and strong evidence in support of a result should be deemed replication-ready. We introduce a measure of replication distance between studies to sensibly study the effect of replication-readiness on scientific inference. We define this measure of distance between an original study and its replications based on results reproducibility. We illustrate how practitioners can benefit from the use of replication distance in exploring the study space before designing their next study with an example.