When Do Multi-Agent Systems Outperform? Analysing the Learning Efficiency of Agentic Systems

Reinforcement Learning (RL) has emerged as a crucial method for training or fine-tuning large language models (LLMs), enabling adaptive, task-specific optimizations through interactive feedback. Multi-Agent Reinforcement Learning (MARL), in particular, offers a promising avenue by decomposing complex tasks into specialized subtasks learned by distinct interacting agents, potentially enhancing the ability and efficiency of LLM systems. However, theoretical insights regarding when and why MARL outperforms Single-Agent RL (SARL) remain limited, creating uncertainty in selecting the appropriate RL framework. In this paper, we address this critical gap by rigorously analyzing the comparative sample efficiency of MARL and SARL within the context of LLM. Leveraging the Probably Approximately Correct (PAC) framework, we formally define SARL and MARL setups for LLMs, derive explicit sample complexity bounds, and systematically characterize how task decomposition and alignment influence learning efficiency. Our results demonstrate that MARL improves sample complexity when tasks naturally decompose into independent subtasks, whereas dependent subtasks diminish MARL’s comparative advantage. Additionally, we introduce and analyze the concept of task alignment, quantifying the trade-offs when enforcing independent task decomposition despite potential misalignments. These theoretical insights clarify empirical inconsistencies and provide practical criteria for deploying MARL strategies effectively in complex LLM scenarios.

💡 Research Summary

**
This paper tackles a fundamental question in the emerging field of agentic large language models (LLMs): under what conditions does multi‑agent reinforcement learning (MARL) provide a genuine sample‑efficiency advantage over the more traditional single‑agent reinforcement learning (SARL) approach? To answer this, the authors adopt the Probably Approximately Correct (PAC) learning framework, which allows them to derive explicit sample‑complexity bounds for both paradigms in the context of sequence‑to‑sequence LLM tasks.

The SARL setting is formalized as a single policy πθ that, given a prompt x, generates an entire output sequence y and receives a scalar reward R(x, y) only after the full sequence is produced. The policy class Π is assumed to have effective dimension d (e.g., the number of trainable LoRA parameters). Using standard PAC arguments, Theorem 4.1 shows that to learn an ε‑optimal policy with confidence 1 − δ, SARL requires O((d/ε²)·log(1/δ)) samples.

In the MARL formulation, the output is partitioned into K contiguous segments y(1)…y(K). Each segment is generated by a dedicated agent πθ_i that observes the original prompt and all previously generated segments. The authors consider two reward‑decomposition models:

1. Dependent reward R_dep = (1/K)∑ r_i(x, y(i), y(<i)), where each sub‑reward may depend on earlier segments, capturing realistic inter‑step dependencies.
2. Independent reward R_indep = (1/K)∑ r_i(x, y(i)), which assumes perfect independence between subtasks.

Theorem 4.2 (dependent case) proves that the MARL sample complexity scales with the sum of the effective dimensions of all agents, i.e., O((∑ d_i / ε²)·log(1/δ)). This reflects the intuition that when subtasks are tightly coupled, learning each one contributes cumulatively to the overall difficulty. Conversely, Theorem 4.3 (independent case) shows that the complexity is dominated by the hardest subtask alone: O((max_i d_i / ε²)·log(1/δ)). Thus, if a task naturally decomposes into truly independent components, MARL can dramatically reduce the number of required interactions compared with SARL.

A novel contribution is the formalization of task alignment. In practice, designers may enforce an independent decomposition even when the true reward exhibits some dependence. The authors define an alignment error Δ = max_x |R_dep(x, y) − R_indep(x, y)| and prove in Theorem 4.6 that this error adds an additive term O(Δ/ε²) to the MARL sample bound. Proposition 4.7 further characterizes a regime where Δ is sufficiently small that MARL still outperforms SARL despite the mis‑alignment. This result explains why some empirical studies report MARL benefits even when the underlying task is not perfectly separable.

Beyond the theorems, the paper provides practical guidance. It suggests measuring task independence (e.g., via mutual information or conditional entropy between segments) and estimating the alignment error before committing to a MARL architecture. If independence is high and alignment error low, the practitioner can expect a sample‑efficiency gain; otherwise, a single‑agent approach may be safer.

The authors acknowledge several limitations. Their analysis assumes (a) a finite‑dimensional policy class (common in LoRA‑style fine‑tuning), (b) delayed, scalar rewards only at the end of the episode, and (c) a turn‑taking, sequential execution model. Real‑world LLM deployments often involve dense or intermediate rewards, asynchronous agent interactions, and full‑model fine‑tuning, which are not covered. Future work is outlined to extend PAC analysis to continuous‑reward MARL, non‑sequential collaboration, and high‑dimensional parameter spaces.

In summary, this work delivers the first rigorous PAC‑based comparison of SARL and MARL for LLM‑driven tasks, pinpointing task decomposition independence and reward alignment as the decisive factors governing sample‑efficiency advantages. The derived bounds reconcile previously contradictory empirical findings and furnish a clear, quantitative decision‑making framework for researchers and engineers contemplating multi‑agent designs in large‑scale language model training and fine‑tuning.