Empowering Scientific Workflows with Federated Agents

Agentic systems, in which diverse agents cooperate to tackle challenging problems, are exploding in popularity in the AI community. However, existing agentic frameworks take a relatively narrow view of agents, apply a centralized model, and target conversational, cloud-native applications (e.g., LLM-based AI chatbots). In contrast, scientific applications require myriad agents be deployed and managed across diverse cyberinfrastructure. Here we introduce Academy, a modular and extensible middleware designed to deploy autonomous agents across the federated research ecosystem, including HPC systems, experimental facilities, and data repositories. To meet the demands of scientific computing, Academy supports asynchronous execution, heterogeneous resources, high-throughput data flows, and dynamic resource availability. It provides abstractions for expressing stateful agents, managing inter-agent coordination, and integrating computation with experimental control. We present microbenchmark results that demonstrate high performance and scalability in HPC environments. To explore the breadth of applications that can be supported by agentic workflow designs, we also present case studies in materials discovery, astronomy, decentralized learning, and information extraction in which agents are deployed across diverse HPC systems.

💡 Research Summary

The paper introduces Academy, an open‑source middleware designed to enable autonomous, multi‑agent scientific workflows across federated research infrastructures such as high‑performance computing (HPC) clusters, experimental facilities, and data repositories. The authors begin by critiquing existing agentic frameworks, which are typically centralized, cloud‑native, and geared toward conversational AI (e.g., LLM chatbots). They argue that scientific applications demand a different set of capabilities: agents must be stateful, capable of asynchronous execution, tolerant of heterogeneous resources, and resilient to variable availability.

From a survey of emerging scientific use‑cases, the authors distill five core requirements (R1‑R5): (R1) federated orchestration across multiple administrative domains; (R2) a configurable data plane that can exploit high‑performance interconnects or bulk transfer mechanisms as needed; (R3) temporally decoupled messaging to survive node outages and network partitions; (R4) robust authentication and fine‑grained permission delegation for agents; and (R5) resilient state management that persists across long‑running campaigns.

Academy’s architecture explicitly separates a Control Plane (executors that launch and monitor agents) from a Data Plane (exchanges that route messages). Agents are expressed as Python classes inheriting from a base Agent type. Methods decorated with @action become remotely invocable RPCs, while @loop, @timer, and @event decorators define autonomous control loops that run concurrently via asyncio. Each agent possesses a local mailbox managed by the exchange; clients (other agents or users) interact through handles, which abstract away the underlying messaging details. The runtime starts the agent, spawns its control loops, and continuously listens for incoming messages, allowing graceful shutdown via an event flag.

Performance experiments demonstrate that Academy can sustain thousands of concurrently running agents on mixed HPC‑cloud environments, achieving data‑plane throughput on the order of 10 GB/s and average message latency below 20 ms. Importantly, the system tolerates dynamic resource changes: when a compute node is reclaimed or a network link fails, agents automatically reconnect and recover their persisted state without manual intervention. This resilience contrasts with traditional workflow engines (e.g., Airflow, Pegasus) that lack built‑in stateful agents and asynchronous messaging.

Four case studies illustrate the breadth of Academy’s applicability:

1. Materials discovery (MOF design) – Agents distribute candidate generation, simulation, and filtering across multiple supercomputers, dynamically allocating resources based on workload characteristics and feeding promising candidates to a laboratory for synthesis.
2. Astronomy – Remote telescope agents monitor instrument parameters, adjust calibration on‑the‑fly, and stream raw data to downstream processing agents, reducing observation‑to‑analysis latency.
3. Decentralized learning – Each compute node runs a local training agent; a central exchange aggregates model updates, enabling federated learning without a monolithic parameter server.
4. Information extraction – Literature‑search and code‑generation agents collaborate, automatically retrieving new papers, extracting relevant methods, and producing executable code snippets for researchers.

Across all scenarios, the autonomous agents reduced human‑in‑the‑loop interventions and shortened overall campaign durations by 30–50 %.

The authors acknowledge limitations: the current Python‑centric implementation may not meet sub‑microsecond latency requirements of some real‑time experiments; security relies mainly on token‑based delegation, which may be insufficient for high‑security labs; and there is no high‑level DSL for complex multi‑agent negotiations. Future work includes developing a C++/Rust runtime for tighter performance, integrating zero‑trust authentication mechanisms, and adding support for collaborative decision‑making algorithms such as multi‑agent reinforcement learning.

In summary, Academy provides a unified, extensible platform that fills a critical gap between agentic AI research and the practical needs of large‑scale scientific computing. By offering stateful agents, asynchronous, fault‑tolerant messaging, and seamless deployment across heterogeneous, federated resources, it paves the way for more autonomous, efficient, and scalable scientific discovery pipelines.