Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics
Abel Brodeur, Nikolai Cook, Anthony Heyes
Abstract
The credibility revolution in economics has promoted causal identification using randomized control trials (RCT), difference-in-differences (DID), instrumental variables (IV) and regression discontinuity design (RDD). Applying multiple approaches to over 21,000 hypothesis tests published in 25 leading economics journals, we find that the extent of p-hacking and publication bias varies greatly by method. IV (and to a lesser extent DID) are particularly problematic. We find no evidence that (i) papers published in the Top 5 journals are different to others; (ii) the journal “revise and resubmit” process mitigates the problem; (iii) things are improving through time. (JEL A14, C12, C52)
Topics & Concepts
CredibilityRegression discontinuity designInstrumental variablePublication biasHackerCausal inferenceOmitted-variable biasEconomicsIdentification (biology)EconometricsRandomized experimentSelection biasPsychologyStatisticsPolitical scienceComputer scienceLawMEDLINEMathematicsComputer securityBiologyBotanyAdvanced Causal Inference Techniques