ALIGN: Aligned Delegation with Performance Guarantees for Multi-Agent LLM Reasoning

LLMs often underperform on complex reasoning tasks when relying on a single generation-and-selection pipeline. Inference-time ensemble methods can improve performance by sampling diverse reasoning paths or aggregating multiple candidate answers, but they typically treat candidates independently and provide no formal guarantees that ensembling improves reasoning quality. We propose a novel method, Aligned Delegation for Multi-Agent LLM Reasoning (ALIGN), which formulates LLM reasoning as an aligned delegation game. In ALIGN, a principal delegates a task to multiple agents that generate candidate solutions under designed incentives, and then selects among their outputs to produce a final answer. This formulation induces structured interaction among agents while preserving alignment between agent and principal objectives. We establish theoretical guarantees showing that, under a fair comparison with equal access to candidate solutions, ALIGN provably improves expected performance over single-agent generation. Our analysis accommodates correlated candidate answers and relaxes independence assumptions that are commonly used in prior work. Empirical results across a broad range of LLM reasoning benchmarks consistently demonstrate that ALIGN outperforms strong single-agent and ensemble baselines.

💡 Research Summary

The paper introduces ALIGN (Aligned Delegation for Multi‑Agent LLM Reasoning), a game‑theoretic framework that treats the inference‑time reasoning process as an aligned delegation game between a principal (acting as a proxy for user preferences) and multiple LLM agents. Each agent independently generates a set of candidate answers, evaluates them with its own internal utility (derived from self‑consistency or similar heuristics), and then selects a single answer to submit according to a stochastic policy. The principal receives all submitted answers, ranks them using a global utility function U that reflects the true task objective, and provides scalar feedback r_i to each agent based on its relative rank. An agent’s overall reward is the product r_i·U_yi, combining the principal’s ranking signal with its internal confidence.

To adapt policies, the authors employ online mirror descent, which in this setting reduces to an exponential‑weight update (the Hedge algorithm). After each round, agents update utility estimates for all their candidates and recompute their selection probabilities proportionally to exp(η·U_t(a)). This iterative process balances exploration of diverse reasoning paths with exploitation of high‑utility answers, and it converges to a Nash equilibrium with sub‑linear regret.

The theoretical contribution rests on three core assumptions: (i) Pareto‑optimal play (agents never submit an answer that is strictly worse for both themselves and the principal), (ii) symmetry among agents (identical number of samples and shared distribution D), and (iii) non‑negative correlation between the principal’s utility and each agent’s internal utility. Under these conditions, the authors prove two main results. First, for any single‑agent mechanism M, there exists a multi‑agent mechanism M′ that, at equilibrium, yields at least as much expected utility for the principal, even when both settings have equal total access to candidate samples. Second, if agents are willing to tolerate a utility loss of at most 2ε relative to their personal optimum, a (2ε)-approximate Bayes‑Nash equilibrium exists such that the principal’s expected utility equals the optimal achievable utility E