Deciding what to replicate: A decision model for replication study selection under resource and knowledge constraints

Robust scientific knowledge is contingent upon replication of original findings. However,researchers who conduct replication studies face a dicult problem; there are many morestudies in need of replication than there are funds available for replicating. To select studies forreplication eciently, we need to understand which studies arethe mostin need of replication.In other words, we need to understand which replication eorts have the highest expected utility.In this article we propose a general rule for study selection in replication research based onthereplication valueof the claims considered for replication. Thereplication valueof a claimis defined as the maximum expected utility we could gain by conducting a replication of theclaim, and is a function of (1) the value of being certain about the claim, and (2) uncertaintyabout the claim based on current evidence. We formalize this definition in terms of a causaldecision model, utilizing concepts from decision theory and causal graph modeling. We discussthe validity of usingreplication valueas a measure of expected utility gain, and we suggestapproaches for deriving quantitative estimates ofreplication value.