Projective independence tests in high dimensions: the curses and the cures
Zhang Yao-wu, Liping Zhu
Abstract
Summary Testing independence between high-dimensional random vectors is fundamentally different from testing independence between univariate random variables. Taking the projection correlation as an example, it suffers from at least three problems. First, it has a high computational complexity of O{n3(p+q)}, where n, p and q are the sample size and dimensions of the random vectors; this limits its usefulness substantially when n is extremely large. Second, the asymptotic null distribution of the projection correlation test is rarely tractable; therefore, random permutations are often suggested as a means of approximating the asymptotic null distribution, which further increases the complexity of implementing independence tests. Third, the power performance of the projection correlation test deteriorates in high dimensions. To address these issues, the projection correlation is improved by using a modified weight function, which reduces the complexity to O{n2(p+q)}. We estimate the improved projection correlation with U-statistic theory. Importantly, its asymptotic null distribution is standard normal, thanks to the high dimesnionality of the random vectors. This expedites the implementation of independence tests substantially. To enhance the power performance in high dimensions, we propose incorporating a cross-validation procedure with feature screening into the projection correlation test. The implementation efficacy and power enhancement are confirmed through extensive numerical studies.