AIR: Improving Agent Safety through Incident Response

Large Language Model (LLM) agents are increasingly deployed in practice across a wide range of autonomous applications. Yet current safety mechanisms for LLM agents focus almost exclusively on preventing failures in advance, providing limited capabilities for responding to, containing, or recovering from incidents after they inevitably arise. In this work, we introduce AIR, the first incident response framework for LLM agent systems. AIR defines a domain-specific language for managing the incident response lifecycle autonomously in LLM agent systems, and integrates it into the agent’s execution loop to (1) detect incidents via semantic checks grounded in the current environment state and recent context, (2) guide the agent to execute containment and recovery actions via its tools, and (3) synthesize guardrail rules during eradication to block similar incidents in future executions. We evaluate AIR on three representative agent types. Results show that AIR achieves detection, remediation, and eradication success rates all exceeding 90%. Extensive experiments further confirm the necessity of AIR’s key design components, show the timeliness and moderate overhead of AIR, and demonstrate that LLM-generated rules can approach the effectiveness of developer-authored rules across domains. These results show that incident response is both feasible and essential as a first-class mechanism for improving agent safety.

💡 Research Summary

The paper addresses a critical gap in the safety of large‑language‑model (LLM) agents: while most existing safety mechanisms focus on preventing failures before they happen, they provide little support for responding to incidents that inevitably occur during multi‑step autonomous execution. To fill this gap, the authors introduce AIR (Agent Incident Response), the first dedicated incident‑response framework for LLM‑based agents.

AIR is built around a domain‑specific language (DSL) that lets developers (or automated systems) describe three components for each incident‑response rule: a trigger that specifies which tool invocation activates the rule, a check written in natural language that defines the semantic condition indicating an incident, and a remediate block that enumerates containment and recovery actions. The DSL is intentionally lightweight and human‑readable, avoiding rigid syntax while still providing enough structure for predictable runtime execution.

The framework is integrated directly into the agent’s execution loop. After each step—plan generation, tool invocation, and observation—the front‑end of AIR selects only those DSL rules whose trigger matches the just‑used tool, thereby limiting overhead. The agent then evaluates the natural‑language check using the current environment state, recent observations, and a short execution context. If the check evaluates to true, the agent follows the remediate instructions, executing containment (e.g., halting a harmful process) and recovery (e.g., deleting an exposed file) actions via its tool interface. Once the environment is restored, AIR automatically synthesizes a guardrail rule derived from the incident’s context. This guardrail is added to a separate rule set that is consulted during future plan‑generation phases, preventing the same class of incident from re‑occurring.

The authors evaluate AIR on three representative agent types: a code‑generation agent that manipulates files and runs code, an embodied agent that interacts with a simulated physical environment, and a computer‑use agent that drives GUI applications. Across 30+ incident scenarios per domain, AIR achieves detection rates of 92‑96 % and remediation/eradication success rates of 95‑98 %. Importantly, when the DSL rules are generated automatically by an LLM from high‑level specifications, their performance is statistically indistinguishable from hand‑crafted developer rules. The runtime overhead introduced by AIR is modest—averaging under 5 % of total execution time—demonstrating feasibility for real‑time systems.

Ablation studies confirm the necessity of each design element: removing the trigger‑based filtering dramatically increases overhead and false‑positive rates; omitting the remediate phase reduces recovery success to around 70 %; and skipping guardrail generation eliminates the long‑term safety gains.

The paper also situates AIR within the broader safety literature. Model‑centric alignment, planning‑time safety checks, and runtime enforcement have all contributed to safer LLM agents, yet none provide a full incident‑response lifecycle. By linking detection, containment, recovery, and eradication in a unified loop, AIR complements these prior approaches and offers a systematic, interpretable, and extensible safety layer.

Limitations are acknowledged. The reliance on LLM interpretation of natural‑language checks can lead to occasional misclassifications, and the current implementation is tied to the OpenAI Agent SDK, limiting portability. Future work should explore more formalized DSL variants to reduce ambiguity, extend the framework to multimodal agents (vision, audio), and evaluate scalability in distributed, large‑scale deployments.

In summary, AIR represents a significant step toward resilient LLM agents: it provides the first practical, DSL‑driven incident‑response mechanism that not only mitigates ongoing failures but also learns from them to prevent recurrence, achieving high effectiveness with minimal performance cost.