DFlash: Block Diffusion for Flash Speculative Decoding

Autoregressive large language models (LLMs) deliver strong performance but require inherently sequential decoding, leading to high inference latency and poor GPU utilization. Speculative decoding mitigates this bottleneck by using a fast draft model whose outputs are verified in parallel by the target LLM; however, existing methods still rely on autoregressive drafting, which remains sequential and limits practical speedups. Diffusion LLMs offer a promising alternative by enabling parallel generation, but current diffusion models typically underperform compared with autoregressive models. In this paper, we introduce DFlash, a speculative decoding framework that employs a lightweight block diffusion model for parallel drafting. By generating draft tokens in a single forward pass and conditioning the draft model on context features extracted from the target model, DFlash enables efficient drafting with high-quality outputs and higher acceptance rates. Experiments show that DFlash achieves over 6x lossless acceleration across a range of models and tasks, delivering up to 2.5x higher speedup than the state-of-the-art speculative decoding method EAGLE-3.

💡 Research Summary

DFlash introduces a novel speculative decoding framework that leverages a lightweight block diffusion model to accelerate inference of large language models (LLMs) while preserving lossless output quality. Traditional speculative decoding methods such as EAGLE‑1, EAGLE‑2, EAGLE‑3, and Medusa rely on an autoregressive draft model to propose future token sequences. Because the draft model must generate tokens sequentially, its drafting latency (T_draft) grows linearly with the speculation budget γ, forcing the draft model to be shallow and limiting its ability to produce high‑quality drafts. Consequently, the expected acceptance length τ quickly saturates, capping practical speedups at roughly 2–3×.

DFlash replaces the autoregressive draft with a block diffusion model that can denoise an entire block of masked tokens in a single forward pass. This parallel generation makes T_draft approximately equal to a single parallel latency t_parallel, which is largely independent of γ. As a result, deeper and more expressive diffusion drafts can be employed without incurring additional latency, breaking the trade‑off that constrains conventional approaches.

The key innovation lies in conditioning the diffusion draft on rich contextual features extracted from the frozen target LLM. During the initial “prefill” step, the target model generates the first token and simultaneously outputs hidden states from a set of uniformly sampled layers ranging from shallow to deep. These hidden states are concatenated, projected into a compact context vector, and injected directly into the Key and Value projections of every draft layer’s KV cache. By persisting this fused context throughout the draft network, DFlash ensures that the draft model continuously benefits from the target’s long‑range dependencies and task‑specific semantics, rather than receiving diluted information only at the input level as in prior work.

Training is tailored to the speculative decoding scenario. DFlash constructs blocks by selecting an “anchor” token (which corresponds to the bonus token produced by the target during verification) and masking the remaining positions within the block. Random anchor sampling exposes the draft model to a wide variety of target contexts, improving data efficiency and generalization. All blocks are concatenated into a single sequence and processed in one batch, allowing the GPU to fully exploit parallelism. The diffusion model is trained to align its block‑level predictions with the outputs of the frozen target, ensuring that during inference the draft’s predictions are already highly compatible with the target’s distribution.

Empirical evaluation spans multiple target models (e.g., Qwen‑3‑8B, LLaMA‑2‑7B) and a diverse set of benchmarks, including GSM8K, HumanEval, MBPP, MT‑Bench, and AIME25. With a five‑layer diffusion draft generating 16‑token blocks, DFlash achieves a drafting latency 3–5× lower than a one‑layer autoregressive draft (EAGLE‑3). Overall inference speedups reach up to 6.1× on Qwen‑3‑8B, and across most benchmarks DFlash is on average 2.5× faster than the state‑of‑the‑art EAGLE‑3 while maintaining comparable accuracy, coding success rates, and reasoning performance. Memory overhead remains modest (≈1–2 GB extra), making the approach viable for production serving stacks such as SGLang.

Ablation studies demonstrate that (1) KV‑injection of target features yields substantially longer acceptance lengths than simple feature concatenation, (2) the random anchor‑masking strategy improves both speedup and quality, and (3) increasing the depth of the diffusion draft continues to improve acceptance without a proportional increase in latency, confirming the theoretical advantage of parallel drafting.

In summary, DFlash unifies three complementary ideas: (i) block diffusion for truly parallel draft generation, (ii) persistent, high‑fidelity conditioning on the target model’s hidden representations via KV‑injection, and (iii) seamless integration into the speculative decoding pipeline. This combination overcomes the inherent sequential bottleneck of autoregressive drafting, delivering lossless, high‑throughput LLM inference that is both memory‑efficient and ready for real‑world deployment.