AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference

Large language models (LLMs) have achieved remarkable performance across a wide range of tasks, but their increasing parameter sizes significantly slow down inference. Speculative decoding mitigates this issue by leveraging a smaller draft model to predict candidate tokens, which are then verified by a larger target model. However, existing approaches often require additional training, extensive hyperparameter tuning, or prior analysis of models and tasks before deployment. In this paper, we propose Adaptive Speculative Decoding (AdaSD), a hyperparameter-free decoding scheme that dynamically adjusts generation length and acceptance criteria during inference. AdaSD introduces two adaptive thresholds: one to determine when to stop candidate token generation and another to decide token acceptance, both updated in real time based on token entropy and Jensen-Shannon distance. This approach eliminates the need for pre-analysis or fine-tuning and is compatible with off-the-shelf models. Experiments on benchmark datasets demonstrate that AdaSD achieves up to 49% speedup over standard speculative decoding while limiting accuracy degradation to under 2%, making it a practical solution for efficient and adaptive LLM inference.

💡 Research Summary

This paper introduces AdaSD (Adaptive Speculative Decoding), a novel method designed to accelerate the inference of large language models (LLMs) while addressing key limitations of existing speculative decoding techniques.

The core challenge stems from the memory-bound nature of modern LLMs: each autoregressive generation step requires loading the entire massive set of model weights, creating a significant latency bottleneck. Speculative decoding tackles this by employing a smaller, faster “draft” model to propose a sequence of candidate tokens. These candidates are then verified in parallel by the larger, accurate “target” model, which accepts a valid prefix. This allows the target model’s expensive forward pass to process multiple tokens at once, improving throughput.

However, prior approaches often require additional fine-tuning to align the draft model, extensive hyperparameter search to set fixed candidate lengths or acceptance criteria, or pre-analysis of specific model-task pairs. AdaSD eliminates these requirements by proposing a hyperparameter-free decoding scheme.

The innovation of AdaSD lies in two adaptive thresholds that are dynamically adjusted during inference itself:

1. A Generation Threshold: This determines when the draft model should stop generating candidate tokens. It is based on the entropy of the draft model’s token distribution at each step. High entropy indicates high uncertainty; AdaSD uses this signal to stop generation early, preventing the draft model from wasting time on low-confidence, likely-to-be-rejected tokens.
2. A Verification Threshold: This decides whether a candidate token proposed by the draft model should be accepted by the target model. Instead of relying on strict matching or complex sampling, AdaSD uses the Jensen-Shannon (JS) distance between the probability distributions of the draft and target models. JS distance is bounded between 0 and 1 and satisfies the properties of a true metric, making it a stable and interpretable measure for setting a robust acceptance criterion.

Crucially, these thresholds are not static. AdaSD implements a heuristic feedback mechanism that continuously updates both thresholds in real-time based on statistics (e.g., running averages) computed from the entropy and JS distance of previously generated tokens. This allows the system to automatically adapt to the varying predictability of different parts of the text (e.g., formulaic vs. creative passages) without any manual intervention or pre-configuration.

The authors provide empirical motivation for their design choices. An analysis using Llama 3.1 8B (draft) and 70B (target) models shows a clear separation: accepted tokens consistently exhibit lower draft entropy and lower JS distance compared to rejected tokens, validating both measures as effective signals.

Experiments on benchmark datasets like Alpaca, GSM8K, and HumanEval demonstrate that AdaSD achieves speedups of up to 49% over standard speculative decoding baselines. Importantly, it maintains output quality, limiting task accuracy degradation to under 2% across all tests. A key advantage is its compatibility with off-the-shelf models; it requires no architectural changes, additional training, or task-specific tuning.

In summary, AdaSD presents a practical, adaptive, and efficient solution for LLM inference. By dynamically controlling both the draft generation length and the token acceptance criteria using principled information-theoretic measures, it achieves significant speedups while preserving accuracy, all without the burden of hyperparameter optimization.