Centauri: Enabling Efficient Scheduling for Communication-Computation Overlap in Large Model Training via Communication Partitioning
Chang Chen, Xiuhong Li, Qianchao Zhu, Jiangfei Duan, Peng Sun, Xingcheng Zhang, Chao Yang
Abstract
Efficiently training large language models (LLMs) necessitates the adoption of hybrid parallel methods, integrating multiple communications collectives within distributed partitioned graphs. Overcoming communication bottlenecks is crucial and is often achieved through communication and computation overlaps. However, existing overlap methodologies tend to lean towards either fine-grained kernel fusion or limited operation scheduling, constraining performance optimization in heterogeneous training environments.
Topics & Concepts
Computer scienceScheduling (production processes)ComputationTraining (meteorology)Distributed computingProcessor schedulingComputer networkAlgorithmMathematical optimizationResource (disambiguation)MeteorologyPhysicsMathematicsParallel Computing and Optimization TechniquesAdvanced Neural Network ApplicationsMachine Learning and Algorithms