TriSpec: Ternary Speculative Decoding via Lightweight Proxy Verification

Inference efficiency in Large Language Models (LLMs) is fundamentally limited by their serial, autoregressive generation, especially as reasoning becomes a key capability and response sequences grow longer. Speculative decoding (SD) offers a powerful solution, providing significant speed-ups through its lightweight drafting and parallel verification mechanism. While existing work has nearly saturated improvements in draft effectiveness and efficiency, this paper advances SD from a new yet critical perspective: the verification cost. We propose TriSpec, a novel ternary SD framework that, at its core, introduces a lightweight proxy to significantly reduce computational cost by approving easily verifiable draft sequences and engaging the full target model only when encountering uncertain tokens. TriSpec can be integrated with state-of-the-art SD methods like EAGLE-3 to further reduce verification costs, achieving greater acceleration. Extensive experiments on the Qwen3 and DeepSeek-R1-Distill-Qwen/LLaMA families show that TriSpec achieves up to 35% speedup over standard SD, with up to 50% fewer target model invocations while maintaining comparable accuracy.

💡 Research Summary

**
The paper tackles a fundamental bottleneck in large language model (LLM) inference: the serial nature of autoregressive generation, which becomes especially costly when models are used for reasoning tasks that require long output sequences. Speculative decoding (SD) mitigates this by having a fast “draft” model generate multiple candidate tokens, which are then verified in parallel by the large target model. While recent work has largely saturated improvements in draft speed and acceptance rate, the verification step (the time the target model spends checking the draft) remains a dominant source of latency.

TriSpec introduces a third component—a lightweight proxy verifier—to off‑load most of the verification work from the expensive target model. The proxy is a much smaller model from the same family as the target (e.g., a 1.7 B Qwen3 model serving as a proxy for a 32 B Qwen3 target). Empirical analysis on the ShareGPT dataset shows that the proxy aligns with the target on 82 % of tokens (exact match) and only 6 % of tokens are deemed unacceptable, indicating strong token‑level agreement.

The key to safely delegating verification to the proxy is a margin‑based confidence criterion. For each token the proxy predicts, the difference between its top‑1 and top‑2 probabilities is computed. If this margin exceeds a tunable threshold λ (e.g., 0.5), the proxy’s decision is considered “trusted”; otherwise it is marked “untrusted”. This simple metric cleanly separates tokens that the proxy can verify reliably from those that require the target’s authority.

TriSpec’s routing logic works as follows. After the draft model proposes a block of k tokens, the proxy runs a single parallel pass to produce its own predictions and probability distributions. Two lengths are computed: (1) τ_a, the number of consecutive draft tokens that the proxy accepts based on its own probability ratio; and (2) τ_m, the longest prefix for which the margin criterion holds. If τ_a < τ_m, the proxy’s verification is trusted up to the first rejection point, and the system can accept the draft tokens and locally correct the next token using the proxy’s output—no target model call is needed. If τ_a ≥ τ_m, the proxy is deemed untrusted before the first rejection; the system then forwards the remaining draft tokens (starting at position τ_m + 1) to the target model for authoritative verification. This adaptive, token‑level routing reduces the average number of target model invocations by more than 50 % while keeping the overall acceptance rate high.

The draft component uses the state‑of‑the‑art EA‑GLE‑3 single‑layer architecture, which already minimizes drafting latency and maximizes acceptance length τ. To enable the draft model to consume features from either the target or the proxy, a lightweight one‑layer MLP adapter is added. The authors explore two training regimes: (i) joint training of the draft model and adapter using both proxy and target features, and (ii) freezing a pretrained draft model and fine‑tuning only the adapter on proxy features. Both approaches yield competitive results, with joint training offering a modest boost in acceptance length.

Extensive experiments are conducted on two families of models—Qwen3‑32B (target) and Qwen3‑1.7B (proxy), as well as DeepSeek‑R1‑Distill‑Qwen/LLaMA variants. Five benchmark suites covering math reasoning (GSM8K, Math500) and code generation (HumanEval, MBPP) are used. Baselines include standard speculative decoding (SD), Speculative Cascade (SpecCascade), and EA‑GLE‑3 without a proxy. Results show that TriSpec combined with EA‑GLE‑3 achieves up to 4.18× speedup (average ≈3.5×) over naive decoding, compared with ≈2.8× for EA‑GLE‑3 alone. The proxy‑only configuration (1.7 B model) already yields 2.3×–3.0× speedup but suffers a 10 %+ drop in accuracy; TriSpec recovers most of that loss, keeping accuracy within 0.2 % of the full‑target baseline.

A latency decomposition analysis confirms that the per‑round verification time t_v is dramatically reduced because the proxy’s forward pass is orders of magnitude cheaper than the target’s. The overall latency L ≈ N · (t_d + t_v / τ) therefore drops primarily due to the reduced t_v, while τ remains high thanks to the strong draft model.

The paper’s contributions are threefold: (1) identifying verification cost as a distinct, under‑explored lever for accelerating speculative decoding; (2) proposing a ternary framework that orchestrates a drafter, a lightweight proxy, and the target model with a simple margin‑based routing rule; (3) demonstrating on real‑world LLM families that this approach yields up to 35 % additional speedup over the best existing SD methods and halves the number of expensive target model calls, all with negligible accuracy degradation.

In summary, TriSpec shows that speculative decoding can be made substantially more efficient not only by improving draft generation but also by intelligently delegating verification to a cheap, well‑aligned proxy. This opens a new design space for LLM inference pipelines, especially in latency‑sensitive or resource‑constrained settings such as real‑time services and edge deployments. Future work may explore multi‑proxy hierarchies, dynamic threshold adaptation, or joint training of all three components to further tighten the trade‑off between speed and fidelity.