The Role of Family-wise Error Rate in Determining Statistical Significance
Kristen Nicholson, Matthew Sherman, Srikanth N. Divi, Daniel R. Bowles, Alexander R. Vaccaro
Abstract
The threshold for statistical significance is determined by the maximum allowable probability of Type I error (α). For studies that test multiple hypotheses or make multiple comparisons, the probability of at least 1 Type I error (family-wise error rate; FWER) increases as the number of hypotheses/comparisons increase. It is generally best practice to set the acceptable threshold for FWER to be less than or equal to α. Bonferroni correction and Tukey honestly significant difference test are 2 of the more common methods to control for FWER. When doing exploratory analysis or evaluating secondary outcomes of a study, it may not be necessary or desirable to control for FWER, which reduces the power of the study. However, deciding to control for FWER should be decided during the design of the study.