T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge
Jianyu Wei, Shijie Cao, Ting Cao, Lingxiao Ma, Lei Wang, Yanyong Zhang, Mao Yang
Abstract
The deployment of Large Language Models (LLMs) on edge devices is increasingly important to enhance on-device intelligence. Weight quantization is crucial for reducing the memory footprint of LLMs on devices. However, low-bit LLMs necessitate mixed precision matrix multiplication (mpGEMM) of low precision weights and high precision activations during inference. Existing systems, lacking native support for mpGEMM, resort to dequantize weights for high precision computation. Such an indirect way can lead to a significant inference overhead.
Topics & Concepts
Lookup tableComputer scienceTable (database)Software deploymentEnhanced Data Rates for GSM EvolutionBit (key)The RenaissanceParallel computingEmbedded systemComputer hardwareOperating systemComputer networkTelecommunicationsArtDatabaseArt historyAdvanced Data Storage TechnologiesParallel Computing and Optimization TechniquesAdvanced Memory and Neural Computing