CAM: A Causality-based Analysis Framework for Multi-Agent Code Generation Systems

Despite the remarkable success that Multi-Agent Code Generation Systems (MACGS) have achieved, the inherent complexity of multi-agent architectures produces substantial volumes of intermediate outputs. To date, the individual importance of these intermediate outputs to the system correctness remains opaque, which impedes targeted optimization of MACGS designs. To address this challenge, we propose CAM, the first \textbf{C}ausality-based \textbf{A}nalysis framework for \textbf{M}ACGS that systematically quantifies the contribution of different intermediate features for system correctness. By comprehensively categorizing intermediate outputs and systematically simulating realistic errors on intermediate features, we identify the important features for system correctness and aggregate their importance rankings. We conduct extensive empirical analysis on the identified importance rankings. Our analysis reveals intriguing findings: first, we uncover context-dependent features\textemdash features whose importance emerges mainly through interactions with other features, revealing that quality assurance for MACGS should incorporate cross-feature consistency checks; second, we reveal that hybrid backend MACGS with different backend LLMs assigned according to their relative strength achieves up to 7.2% Pass@1 improvement, underscoring hybrid architectures as a promising direction for future MACGS design. We further demonstrate CAM’s practical utility through two applications: (1) failure repair which achieves a 73.3% success rate by optimizing top-3 importance-ranked features and (2) feature pruning that reduces up to 66.8% intermediate token consumption while maintaining generation performance. Our work provides actionable insights for MACGS design and deployment, establishing causality analysis as a powerful approach for understanding and improving MACGS.

💡 Research Summary

Multi‑Agent Code Generation Systems (MACGS) have become a powerful paradigm for automated software development by decomposing programming tasks among specialized agents. While they achieve superior performance over single‑LLM approaches, the sheer volume of intermediate outputs—plans, requirement analyses, algorithm designs, implementation details—creates an opacity problem: it is unclear how each intermediate artifact contributes to the final code correctness. Existing analysis methods (manual inspection, LLM‑based reasoning, all‑but‑one ablations) suffer from scalability, subjectivity, or coarse granularity.

The authors introduce CAM (Causality‑based Analysis for MACGS), the first framework that applies actual causality theory to quantify the causal impact of intermediate features on final code success. CAM first maps heterogeneous intermediate outputs into a structured set of “features” (e.g., problem understanding, programming language choice, algorithm flow, implementation specifics). These features become nodes in a directed acyclic causal graph. Using the three conditions of actual causality (actuality, counterfactual dependence, minimality), the framework defines when a feature is a cause of a successful or failed outcome.

Two technical innovations enable scalable analysis. First, a large‑language‑model‑driven counterfactual intervention engine systematically injects realistic errors into each feature (e.g., syntactic mistakes, inaccurate specifications) and observes the resulting change in Pass@1. Second, the notion of an “influence set” captures the unique error‑propagation dynamics of multi‑agent pipelines, allowing the algorithm to prune irrelevant execution states and dramatically reduce the number of LLM calls (often by orders of magnitude).

Experiments span several representative MACGS (including MetaGPT) across three backend LLMs (GPT‑4o, Qwen‑2.5‑Coder, DeepSeek‑Coder‑V2) and multiple benchmark datasets. Feature importance rankings produced by CAM correlate strongly with human expert annotations (Kendall τ 0.76–0.91), confirming reliability. Two key findings emerge: (1) Context‑dependent features—features whose importance manifests only when combined with other features—appear in 78.8 % of cases, indicating that failures often stem from subtle incompatibilities rather than isolated errors. This suggests that quality assurance must go beyond module‑level checks to include cross‑feature consistency verification. (2) Hybrid backend architectures, where different LLMs are assigned to stages according to their strengths, achieve up to a 7.2 % improvement in Pass@1 compared with uniform backend configurations, highlighting a promising design direction.

CAM’s practical utility is demonstrated through two downstream applications. In failure repair, correcting only the top‑3 most important features yields a 73.3 % success rate, showing that targeted interventions can recover most failures without full re‑execution. In feature pruning, removing low‑importance features reduces intermediate token consumption by up to 66.8 % while preserving or even slightly improving generation performance, offering a concrete path to more efficient MACGS deployments.

Overall, CAM provides a systematic, causality‑grounded methodology for dissecting the intricate relationship between intermediate agent outputs and final code quality. By delivering actionable insights for architecture design, debugging, and resource optimization, it establishes causal analysis as a foundational tool for the next generation of multi‑agent software generation systems.