The roles, challenges, and merits of the p value
Oliver Y. Chén, Julien Bodelet, Raúl G. Saraiva, Huy P. Phan, Junrui Di, Guy Nagels, Tom Schwantje, Hengyi Cao, Jiangtao Gou, Jenna Reinen, Bin Xiong, Bangdong Zhi, Xiaojun Wang, Maarten De Vos
Abstract
Since the 18th century, the p value has been an important part of hypothesis-based scientific investigation. As statistical and data science engines accelerate, questions emerge: to what extent are scientific discoveries based on p values reliable and reproducible? Should one adjust the significance level or find alternatives for the p value? Inspired by these questions and everlasting attempts to address them, here, we provide a systematic examination of the p value from its roles and merits to its misuses and misinterpretations. For the latter, we summarize modest recommendations to handle them. In parallel, we present the Bayesian alternatives for seeking evidence and discuss the pooling of p values from multiple studies and datasets. Overall, we argue that the p value and hypothesis testing form a useful probabilistic decision-making mechanism, facilitating causal inference, feature selection, and predictive modeling, but that the interpretation of the p value must be contextual, considering the scientific question, experimental design, and statistical principles.