TwinPilots: A New Computing Paradigm for GPU-CPU Parallel LLM Inference
Chengye Yu, Tianyu Wang, Zili Shao, Linjie Zhu, Zhou Xu, Song Jiang
Abstract
When trained Large Language Models (LLMs) become available, it is desirable to carry out LLM inferences at the user end with limited resources. A common belief on LLM inference is that GPU is essentially the only meaningful processor as almost all computation is tensor multiplication that GPU excels in. However, this belief and its practice are challenged by the fact that GPU has insufficient memory and runs at a much slower speed due to constantly waiting for data to be loaded from the CPU memory via a slow PCIe bus. This makes the CPU a processor with meaningful computing power that can be leveraged to accelerate the inference.
Topics & Concepts
Computer scienceParallel computingSupercomputerInferenceGeneral-purpose computing on graphics processing unitsComputer architectureComputational scienceOperating systemArtificial intelligenceGraphicsMagnetic confinement fusion researchDistributed and Parallel Computing SystemsAdvanced Data Storage Technologies