MAESTRO: Multi-Agent Evaluation Suite for Testing, Reliability, and Observability

📝 Original Info

• Title: MAESTRO: Multi-Agent Evaluation Suite for Testing, Reliability, and Observability
• ArXiv ID: 2601.00481
• Date: 2026-01-01
• Authors: Tie Ma, Yixi Chen, Vaastav Anand, Alessandro Cornacchia, Amândio R. Faustino, Guanheng Liu, Shan Zhang, Hongbin Luo, Suhaib A. Fahmy, Zafar A. Qazi, Marco Canini

📝 Abstract

Large language model (LLM)-based multi-agent systems (MAS) are rapidly moving from demos to production, yet their dynamic execution makes them stochastic, failure-prone, and difficult to reproduce or debug. Existing benchmarks largely emphasize application-level outcomes (e.g., task success) and provide limited, non-standardized visibility into execution behavior, making controlled, apples-toapples comparisons across heterogeneous MAS architectures challenging. We present MAESTRO, an evaluation suite for the testing, reliability, and observability of LLM-based MAS. MAESTRO standardizes MAS configuration and execution through a unified interface, supports integrating both native and third-party MAS via a repository of examples and lightweight adapters, and exports framework-agnostic execution traces together with system-level signals (e.g., latency, cost, and failures). We instantiate MAESTRO with 12 representative MAS spanning popular agentic frameworks and interaction patterns, and conduct controlled experiments across repeated runs, backend models, and tool configurations. Our case studies show that MAS execut...

📄 Full Content

...(본문 내용이 길어 생략되었습니다. 사이트에서 전문을 확인해 주세요.)