Sharpness-Aware Minimization Improves Language Model Generalization
Dara Bahri, Hossein Mobahi, Yi Tay
2022Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)43 citationsDOIOpen Access PDF
Abstract
The allure of superhuman-level capabilities has led to considerable interest in language models like GPT-3 and T5, wherein the research has, by and large, revolved around new model architectures, training tasks, and loss objectives, along with substantial engineering efforts to scale up model capacity and dataset size. Comparatively little work has been done to improve the generalization of these models through better optimization.
Topics & Concepts
GeneralizationComputer scienceMaxima and minimaLanguage modelMinificationOverhead (engineering)Convergence (economics)Artificial intelligenceNatural languageTraining setMachine learningScale (ratio)Programming languageMathematicsMathematical analysisEconomic growthQuantum mechanicsEconomicsPhysicsTopic ModelingNatural Language Processing TechniquesSpeech Recognition and Synthesis