Over-parametrized deep neural networks minimizing the empirical risk do not generalize well
Michael Köhler, Adam Krzyżak
Abstract
Recently it was shown in several papers that backpropagation is able to find the global minimum of the empirical risk on the training data using over-parametrized deep neural networks. In this paper, a similar result is shown for deep neural networks with the sigmoidal squasher activation function in a regression setting, and a lower bound is presented which proves that these networks do not generalize well on a new data in the sense that networks which minimize the empirical risk do not achieve the optimal minimax rate of convergence for estimation of smooth regression functions.
Topics & Concepts
Artificial neural networkMinimaxMathematicsSigmoid functionBackpropagationRegressionConvergence (economics)Empirical risk minimizationFunction (biology)Deep neural networksApplied mathematicsMathematical optimizationArtificial intelligenceComputer scienceStatisticsEconomicsEconomic growthEvolutionary biologyBiologySparse and Compressive Sensing TechniquesStatistical Methods and InferenceGaussian Processes and Bayesian Inference