Data and Parameter Scaling Laws for Neural Machine Translation
Mitchell A. Gordon, Kevin Duh, Jared Kaplan
2021Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing25 citationsDOIOpen Access PDF
Abstract
We observe that the development crossentropy loss of supervised neural machine translation models scales like a power law with the amount of training data and the number of non-embedding parameters in the model. We discuss some practical implications of these results, such as predicting BLEU achieved by large scale models and predicting the ROI of labeling data in low-resource language pairs.
Topics & Concepts
Machine translationComputer scienceArtificial intelligenceEmbeddingEntropy (arrow of time)Translation (biology)ScalingMachine learningScaling lawTraining setBLEULanguage modelPrinciple of maximum entropyNatural language processingMathematicsGenePhysicsBiochemistryQuantum mechanicsMessenger RNAGeometryChemistryNatural Language Processing TechniquesTopic ModelingSoftware Engineering Research