KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
Amir Gholami, Coleman Hooper, Kurt Keutzer, Sehoon Kim, Michael W. Mahoney, Hiva Mohammadzadeh, Yakun Sophia Shao
Topics & Concepts
Computer scienceCacheContext (archaeology)AlgorithmInferenceQuantization (signal processing)Artificial intelligenceTheoretical computer scienceVector quantizationData miningCaching and Content DeliveryAdvanced Data Storage TechnologiesNetwork Packet Processing and Optimization