Distributed Adaptive Speculative Decoding: Accelerating Large Language Model Inference With Context-Aware Draft Selection

Abstract

Large Language Models (LLMs) have revolutionized natural language processing, yet their autoregressive generation process remains a critical bottleneck for real-time applications. Speculative decoding has emerged as a promising approach to accelerate inference by leveraging smaller draft models to predict future tokens, which are then verified in parallel by the target model. However, existing methods rely on fixed draft models and single-node execution, limiting their effectiveness across diverse input distributions and scalability in production environments. We propose Distributed Adaptive Speculative Decoding (DASD), a novel framework that introduces three key innovations: (1) context-aware draft model selection using a lightweight routing network that dynamically chooses optimal draft models based on input characteristics, (2) asynchronous distributed speculation that pipelines draft generation and target verification across multiple GPUs with fault-tolerant rollback mechanisms, and (3) adaptive speculation length that adjusts the number of speculative tokens based on real-time acceptance rates. Through extensive experiments on NVIDIA A4000 GPUs using Llama-3-8B and multiple draft models, we demonstrate that DASD achieves <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{2. 6 - 2. 8} \times$</tex> speedup compared to standard autoregressive decoding and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{1 5} \boldsymbol{-} \mathbf{2 0} \boldsymbol{\%}$</tex> improvement over fixed-strategy speculative decoding, while maintaining identical output quality. Our approach shows consistent gains across code generation (76 % acceptance), factual QA (72 % acceptance), and conversational tasks (68 % acceptance). The proposed distributed protocol reduces end-to-end latency by 42 % in multi-GPU settings and provides graceful degradation under GPU failures. We release our implementation and comprehensive benchmarks to facilitate future research in efficient LLM serving.

Topics & Concepts