Reply to: Multivariate BWAS can be replicable with moderate sample sizes
Brenden Tervo‐Clemmens, Scott Marek, Roselyne J. Chauvin, Andrew N. Van, Benjamin P. Kay, Timothy O. Laumann, Wesley K. Thompson, Thomas E. Nichols, B.T. Thomas Yeo, Deanna M. Barch, Beatríz Luna, Damien A. Fair, Nico U.F. Dosenbach
Abstract
In our previous study 1 , we documented the effect of sample size on the reproducibility of brain-wide association studies (BWAS) that aim to cross-sectionally relate individual differences in human brain structure (cortical thickness) or function (resting-state functional connectivity (RSFC)) to cognitive or mental health phenotypes. Applying univariate and multivariate methods (for example, support vector regression (SVR)) to three large-scale neuroimaging datasets (total n ≈ 50,000), we found that overall BWAS reproducibility was low for n < 1,000, due to smaller than expected effect sizes. When samples and true effects are small, sampling variability, and/or overfitting can generate ‘statistically significant’ associations that are likely to be reported due to publication bias, but are not reproducible 2 , 3 , 4 , 5 , and we therefore suggested that BWAS should build on recent precedents 6 , 7 and continue to aim for samples in the thousands. In the accompanying Comment, Spisak et al. 8 agree that larger BWAS are better 5 , 9 , but argue that “multivariate BWAS effects in high-quality datasets can be replicable with substantially smaller sample sizes in some cases” ( n = 75–500); this suggestion is made on the basis of analyses of a selected subset of multivariate cognition/RSFC associations with larger effect sizes, using their preferred method (ridge regression with partial correlations) in a demographically more homogeneous, single-site/scanner sample (Human Connectome Project (HCP), n = 1,200, aged 22–35 years).