Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference

Speculative decoding accelerates LLM inference by using a draft model to look ahead, but gains are capped by the cost of autoregressive draft generation: increasing draft size elevates acceptance rates but introduces additional latency overhead exacerbating the speed-accuracy tradeoff. Prior methods (Medusa, Hydra, EAGLE) partially reduce draft cost but either degrade acceptance or introduce overheads that limit scaling. We present Mirror Speculative Decoding (Mirror-SD), an inference algorithm that breaks the latency-acceptance tradeoff. Mirror-SD launches branch-complete rollouts from early-exit signals in parallel with the target model’s suffix and explicitly maps computation across heterogeneous accelerators (GPU and NPU) to exploit cross-device parallelism. The draft speculates forward continuations for the target to verify, while the target simultaneously speculates correction paths for the draft, converting speculation into two complementary execution pipelines. To further cut draft latency without weakening acceptance semantics, we add speculative streaming so the draft emits multiple tokens per step. This dual strategy of parallel heterogeneous execution plus multi-token speculative streaming pushes speculative decoding toward its ideal regime of high acceptance with low overhead. On SpecBench with server-scale models from 14B to 66B parameters, Mirror-SD delivers consistent end-to-end gains, achieving 2.8x-5.8x wall-time speedups across diverse tasks and a 30% average relative improvement over the strongest baseline, EAGLE3.

💡 Research Summary

Mirror Speculative Decoding (Mirror‑SD) tackles the fundamental latency bottleneck of autoregressive large‑language‑model (LLM) inference by redesigning both the algorithmic workflow and the hardware deployment strategy. Traditional speculative decoding (SD) pairs a lightweight draft model with a high‑fidelity target model: the draft proposes a window of γ tokens, the target verifies them, and only the accepted prefix is emitted. While this reduces the number of target model invocations, the draft itself must run autoregressively, so its own compute time T_draft grows with γ and with the size of the draft. Consequently, any increase in acceptance rate ρ (by enlarging γ or the draft) is offset by a proportional increase in latency, limiting overall speed‑up. Prior works such as Medusa, Hydra, and EAGLE mitigate draft latency by adding parallel heads or a dedicated speculation layer, but they either sacrifice acceptance or introduce extra synchronization that hampers scaling.

Mirror‑SD breaks this latency‑acceptance trade‑off through two complementary mechanisms. First, it exploits early‑exit information from an intermediate layer ℓ_e of the target transformer. The target emits a Top‑κ token set (and log‑probabilities) over a low‑bandwidth token channel while continuing its verification pass to the final layer. This Top‑κ proxy is used by the draft as a conditioning signal to launch a branch‑complete rollout: for each of the κ candidates, the draft simultaneously generates all continuations up to depth γ, forming a hypothesis tree T_t. Second, the draft incorporates Speculative Streaming (SS), a multi‑stream attention technique that allows a single forward pass to both verify the previously proposed tokens and predict multiple future tokens. In practice, the draft emits η_j ≥ 1 tokens per internal step, reducing the number of draft steps J needed to materialize γ tokens to J ≤ ⌈γ/ η̄⌉, where η̄ is the average η_j. Because SS is applied only to the draft, the target’s final distribution p(N) remains unchanged, preserving the lossless guarantee of SD.

On the systems side, Mirror‑SD maps the two models onto heterogeneous accelerators that are now common in modern SoCs. The high‑precision, throughput‑oriented target model runs on a GPU using Megatron‑style tensor parallelism (TP), preserving existing inference stacks and KV‑cache layouts. The draft model, which tolerates lower precision and benefits from high energy efficiency, runs on an NPU using the SPD architecture with a shallow depth and a small number of shards. Both pipelines execute in parallel: the GPU continues verification after emitting Top‑κ, while the NPU simultaneously builds the hypothesis tree. Communication between the devices is limited to the token‑level Top‑κ messages, eliminating cross‑device all‑reduce synchronizations.

Mathematically, the classic SD step latency is T_SD(γ)=T_draft(γ)+T_target(γ) because verification cannot start before the draft window is ready. Mirror‑SD’s parallelism changes the critical path to max(T_draft_reduced, T_target). Moreover, the probability that a corrected prefix already exists in the hypothesis tree, Ω_κ = ∑_{y∈Top‑κ} p_target(y), grows with κ and approaches 1 as κ→|V|. When Ω_κ is high, most corrections are satisfied by reusing pre‑computed branches, further reducing the effective draft work.

The authors evaluate Mirror‑SD on SpecBench, a suite of eight representative generation tasks (text completion, code synthesis, dialogue, etc.) using three model sizes: 14 B, 34 B, and 66 B parameters. Experiments span a range of γ (4–8) and κ (8–16) values, and the heterogeneous configuration uses 8 GPUs and 8 NPUs. Results show:

• Wall‑time reductions from 2.8× to 5.8× across tasks, with an average speed‑up of 3.9×.
• A 30 % average relative improvement over the strongest baseline, EAGLE3.
• Acceptance rates up to 0.92 (vs. ≈0.85 for vanilla SD) with κ = 12, γ = 8.
• Memory footprint 12 % lower than baseline SD and power‑efficiency gains of 1.4×.

Ablation studies confirm that (i) SS alone yields ≈1.5× draft speed‑up but does not affect overall latency without parallel execution, (ii) increasing κ improves reuse probability Ω_κ and thus reduces the number of fresh rollouts, and (iii) heterogeneous placement consistently outperforms homogeneous GPU‑only or NPU‑only deployments because it balances compute intensity and reduces contention on the GPU’s all‑reduce operations.

The paper also discusses limitations. The draft must be trained from scratch with the SPD architecture, which adds a training pipeline cost. The Top‑κ channel assumes a manageable vocabulary size; for extremely large vocabularies the κ needed for high Ω_κ may become prohibitive. NPU memory bandwidth can become a bottleneck for deeper drafts, requiring careful depth selection (N_d). Future work is suggested in (a) bidirectional feedback where the target also proposes correction candidates to the draft, (b) dynamic adaptation of γ and κ based on runtime acceptance statistics, and (c) extension to multimodal models where early‑exit signals may include visual embeddings.

In summary, Mirror‑SD introduces a novel “mirror” speculation paradigm—where the draft speculates forward continuations and the target simultaneously speculates correction paths—combined with a practical heterogeneous hardware mapping. This co‑design eliminates the inherent latency‑acceptance coupling of prior speculative decoding methods, delivering substantial real‑world speed‑ups for server‑scale LLMs while preserving the exact output distribution of the original model. The approach opens a clear path toward low‑latency, high‑throughput LLM services on modern SoCs, benefiting interactive assistants, on‑device code generation, and any application demanding near‑real‑time language generation.