System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
Sam Adé Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Reza Yazdani Aminabadi, Shuaiwen Leon Song, Samyam Rajbhandari, Yuxiong He
Abstract
Long sequences are ubiquitous in NLP tasks such as document summarization, machine translation, and dialogue modeling [1]–[9]. Traditional approaches to parallelism, including data parallelism [10]–[12], tensor [13] and pipeline parallelism [14]–[16] struggle to handle sequences that span thousands or even millions of tokens.
Topics & Concepts
Computer scienceSequence (biology)TransformerSequence diagramEngineeringProgramming languageUnified Modeling LanguageElectrical engineeringVoltageSoftwareBiologyGeneticsMagnetic Properties and Applications