Minimax Optimal Estimation of KL Divergence for Continuous Distributions
Puning Zhao, Lifeng Lai
Abstract
Estimating Kullback-Leibler divergence from identical and independently distributed samples is an important problem in various domains. One simple and effective estimator is based on the k nearest neighbor distances between these samples. In this paper, we analyze the convergence rates of the bias and variance of this estimator. Furthermore, we derive a lower bound of the minimax mean square error and show that kNN method is asymptotically rate optimal.
Topics & Concepts
MinimaxEstimatorDivergence (linguistics)MathematicsKullback–Leibler divergenceConvergence (economics)Upper and lower boundsApplied mathematicsMean squared errorMinimax estimatorRate of convergenceStatisticsMathematical optimizationVariance (accounting)Simple (philosophy)Minimum-variance unbiased estimatorComputer scienceMathematical analysisEpistemologyAccountingPhilosophyComputer networkChannel (broadcasting)BusinessLinguisticsEconomic growthEconomicsStatistical Methods and InferenceSparse and Compressive Sensing TechniquesAdvanced Statistical Methods and Models