Large Covariance Matrix Estimation With Oracle Statistical Rate via Majorization-Minimization
Quanmin Wei, Ziping Zhao
Abstract
The <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\ell_1$</tex-math></inline-formula> penalized covariance estimator has been widely used for estimating large sparse covariance matrices. It is recognized that <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\ell_1$</tex-math></inline-formula> penalty introduces a non-negligible estimation bias, while a proper utilization of non-convex penalty may lead to an estimator with a refined statistical rate of convergence. To eliminate the estimation bias, in this paper we propose to estimate large sparse covariance matrices using the non-convex penalty. It is challenging to analyze the theoretical properties of the resulting estimator because popular iterative algorithms for convex optimization no longer have global convergence guarantees for non-convex optimization. To tackle this issue, an efficient algorithm based on the majorization-minimization (MM) framework is developed by solving a sequence of convex relaxation subproblems. An approximation solution to each subproblem is obtained via the proximal gradient method with a linear convergence rate. We clearly establish the statistical properties of all the approximate solutions generated by the MM-based algorithm and prove that the proposed estimator achieves the oracle statistical rate in the Frobenius norm under weak technical assumptions. We also consider a modification of the proposed estimation method using the correlation matrix and show that the modified correlation-based covariance estimator enjoys a better rate in the spectral norm. Our theoretical findings are corroborated through extensive numerical experiments on both synthetic data and real-world datasets.