Liger: Interleaving Intra- and Inter-Operator Parallelism for Distributed Large Model Inference
Jiangsu Du, Jinhui Wei, Jiazhi Jiang, Shenggan Cheng, Dan Huang, Zhiguang Chen, Yutong Lu
Abstract
Distributed large model inference is still in a dilemma where balancing cost and effect. The online scenarios demand intraoperator parallelism to achieve low latency and intensive communications makes it costly. Conversely, the inter-operator parallelism can achieve high throughput with much fewer communications, but it fails to enhance the effectiveness.
Topics & Concepts
Computer scienceInterleavingParallelism (grammar)Latency (audio)InferenceTask parallelismData parallelismParallel computingOperator (biology)ThroughputDilemmaDistributed computingWirelessOperating systemArtificial intelligenceTelecommunicationsRepressorEpistemologyPhilosophyBiochemistryGeneChemistryTranscription factorAdvanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningCOVID-19 diagnosis using AI