Complexity control by gradient descent in deep networks
Tomaso Poggio, Qianli Liao, Andrzej Banburski
Abstract
Overparametrized deep networks predict well, despite the lack of an explicit complexity control during training, such as an explicit regularization term. For exponential-type loss functions, we solve this puzzle by showing an effective regularization effect of gradient descent in terms of the normalized weights that are relevant for classification.
Topics & Concepts
Gradient descentRegularization (linguistics)Computer scienceExponential functionStochastic gradient descentDeep learningArtificial intelligenceAlgorithmApplied mathematicsMathematicsArtificial neural networkMathematical analysisBlind Source Separation TechniquesNeural Networks and ApplicationsMachine Learning and ELM