Understanding Estimation and Generalization Error of Generative Adversarial Networks
Kaiyi Ji, Yi Zhou, Yingbin Liang
Abstract
This article investigates the estimation and generalization errors of the generative adversarial network (GAN) training. On the statistical side, we develop an upper bound as well as a minimax lower bound on the estimation error for training GANs. The upper bound incorporates the roles of both the discriminator and the generator of GANs, and matches the minimax lower bound in terms of the sample size and the norm of the parameter matrices of neural networks under ReLU activation. On the algorithmic side, we develop a generalization error bound for the stochastic gradient method (SGM) in training GANs. Such a bound justifies the generalization ability of the GAN training via SGM after multiple passes over the data and reflects the interplay between the discriminator and the generator. Our results imply that the training of the generator requires more samples than the training of the discriminator. This is consistent with the empirical observation that the training of the discriminator typically converges faster than that of the generator. The experiments validate our theoretical results.