Subjective p intervals researchers underestimate the variability of p values over replication

Abstract

Suppose you obtain p =.02 in an experiment, then replicate the experiment with new samples. What p value might you obtain, and what interval has an 80% chance of including that replication p? Under conservative assumptions the answer is, perhaps surprisingly (.0003,.30). The authors report three email surveys that asked authors of articles published in leading journals in psychology, medicine, or statistics to estimate such intervals. Overall response rate (7%) was low, but responses from 360 researchers gave intervals with an average 40% to 50% chance of including replication p, rather than the target 80%. Results were similar for all three disciplines. Respondents generally found the task ..