SpecFuse: Ensembling Large Language Models via Next-Segment Prediction

Ensembles of generative large language models (LLMs) are a promising way to compensate for individual model limitations, integrating the strengths of different LLMs. Existing LLM ensemble methods, however, face limitations such as first-token delay and challenges in long-range semantic collaboration between models, Moreover, they typically assume equal voting weights for all models during ensemble, ignoring task-specific performance differences among models. In this work, we propose SpecEM, a training-free, plug-and-play LLM ensemble framework that dynamically adjusts each model’s model contribution in real time based on task performance. Inspired by speculative decoding, SpecEM iteratively performs drafting and verification, allowing models to collaborate semantically at the segment level for integrated output. Furthermore, we introduce an online feedback mechanism with multiplicative weight updates, where each model’s voting weight is adjusted on-the-fly according to how often it outperforms others during verification stage, ensuring that stronger models exert greater influence during ensembling. Experimental results on five LLM families (ranging from 7B to 72B parameters) and six benchmark datasets, spanning open-domain instruction following, reasoning, commonsense, demonstrate consistent performance improvements compared to state-of-the-art LLM ensemble methods. Our code is available at https://github.com/lvbotenbest/SpecEM.

💡 Research Summary

The paper introduces SpecEM, a training‑free, plug‑and‑play ensemble framework for large language models (LLMs) that addresses two major shortcomings of existing LLM ensembling methods: first‑token latency and limited long‑range semantic collaboration. Existing approaches fall into two categories. “Generate‑then‑ensemble” methods wait for all models to finish full responses before a selector or fusion model merges them, causing users to wait for the first token. “Ensemble‑while‑generation” methods fuse token‑level probability distributions on the fly, but they suffer from vocabulary mismatches, computational overhead, and poor coordination over long spans.

SpecEM draws inspiration from speculative decoding and restructures ensembling as an iterative drafting‑verification loop. In each iteration (round k), all N base models receive the same prompt and the best segment selected from the previous round. During the drafting stage each model independently generates a candidate segment Cₖⁱ of bounded length L. In the verification stage, every model scores all candidate segments. Scoring is performed by averaging the token‑level logits of a candidate, normalizing per‑model scores, and then aggregating them with model‑specific weights ωₖⁱ to obtain overall candidate scores yₖʲ. The top‑scoring segment becomes the new context for the next drafting round.

A key novelty is the online feedback mechanism that dynamically updates the verification weights ωₖⁱ. The mechanism assumes that a model that frequently produces the best drafts also provides more reliable evaluations. For each round, a reward γₖⁱ is computed as the proportion of pairwise comparisons where model i’s draft outranks another model’s draft according to the remaining validators. The weight update follows a multiplicative rule: ωₖⁱ = ωₖ₋₁ⁱ·exp(η·γₖⁱ), where the learning rate η = α·√(1/k)/N adapts to the number of rounds and models, preventing the initial uniform weight (1/N) from becoming too small. After the exponential update, weights are normalized to sum to one.

To keep verification efficient, SpecEM introduces a “verify‑in‑line” technique. All candidate segments are concatenated into a single sequence, but a custom attention mask blocks attention across different candidates, ensuring each model only attends to the shared prior context and its own candidate. Position IDs are also reset so that each candidate appears immediately after the prior context, preserving the intended positional relationships. This design enables parallel scoring of all candidates without extra forward passes for each model‑candidate pair.

The authors evaluate SpecEM on five LLM families ranging from 7 B to 72 B parameters and six benchmark datasets covering open‑domain instruction following, reasoning, and commonsense tasks. Compared with state‑of‑the‑art ensemble baselines such as MBR, GenFuse, UniTE, and EV‑A, SpecEM consistently yields 1.2–2.5 percentage‑point improvements in accuracy or standard metrics. Ablation studies confirm that (1) dynamic weighting outperforms static equal weighting, (2) the verify‑in‑line mask reduces memory and compute while preserving performance, and (3) the proposed reward formulation is more effective than simpler selection heuristics.

Limitations include sensitivity to the maximum segment length L and the number of iterations k; very long generation tasks may incur higher latency due to more rounds. The current reward definition relies on majority voting, which could suppress a consistently strong model if other models dominate early rounds. Future work is suggested to explore more nuanced reward functions, adaptive segment lengths per round, and richer inter‑model communication mechanisms.

In summary, SpecEM offers a practical, training‑free ensemble strategy that leverages iterative drafting, parallel verification, and online multiplicative weight updates to let heterogeneous LLMs collaborate at the segment level, achieving superior performance without the need for additional fusion models or fine‑tuning.