AdaPipe: Optimizing Pipeline Parallelism with Adaptive Recomputation and Partitioning
Zhenbo Sun, Huanqi Cao, Yuanwei Wang, Guanyu Feng, Shengqi Chen, Haojie Wang, Wenguang Chen
Abstract
Large language models (LLMs) have demonstrated powerful capabilities, requiring huge memory with their increasing sizes and sequence lengths, thus demanding larger parallel systems. The broadly adopted pipeline parallelism introduces even heavier and unbalanced memory consumption. Recomputation is a widely employed technique to mitigate the problem but introduces extra computation overhead.
Topics & Concepts
Computer scienceParallel computingParallelism (grammar)Pipeline (software)Task parallelismInstruction-level parallelismProgramming languageAlgorithms and Data CompressionParallel Computing and Optimization TechniquesEmbedded Systems Design Techniques