Truth or backpropaganda? An empirical investigation of deep learning theory
Micah Goldblum, Jonas Geiping, Avi Schwarzschild, Michael Moeller, Tom Goldstein
2020International Conference on Learning Representations17 citations
Abstract
We empirically evaluate common assumptions about neural networks that are widely held by practitioners and theorists alike. We study the prevalence of local minima in loss landscapes, whether small-norm parameter vectors generalize better (and whether this explains the advantages of weight decay), whether wide-network theories (like the neural tangent kernel) describe the behaviors of classifiers, and whether the rank of weight matrices can be linked to generalization and robustness in real-world networks.
Topics & Concepts
Maxima and minimaRobustness (evolution)Artificial neural networkArtificial intelligenceGeneralizationComputer scienceMachine learningEmpirical researchRank (graph theory)Kernel (algebra)MathematicsStatisticsDiscrete mathematicsCombinatoricsChemistryBiochemistryGeneMathematical analysisAdversarial Robustness in Machine LearningStochastic Gradient Optimization TechniquesDomain Adaptation and Few-Shot Learning