AgenticRed: Optimizing Agentic Systems for Automated Red-teaming

While recent automated red-teaming methods show promise for systematically exposing model vulnerabilities, most existing approaches rely on human-specified workflows. This dependence on manually designed workflows suffers from human biases and makes exploring the broader design space expensive. We introduce AgenticRed, an automated pipeline that leverages LLMs’ in-context learning to iteratively design and refine red-teaming systems without human intervention. Rather than optimizing attacker policies within predefined structures, AgenticRed treats red-teaming as a system design problem. Inspired by methods like Meta Agent Search, we develop a novel procedure for evolving agentic systems using evolutionary selection, and apply it to the problem of automatic red-teaming. Red-teaming systems designed by AgenticRed consistently outperform state-of-the-art approaches, achieving 96% attack success rate (ASR) on Llama-2-7B (36% improvement) and 98% on Llama-3-8B on HarmBench. Our approach exhibits strong transferability to proprietary models, achieving 100% ASR on GPT-3.5-Turbo and GPT-4o, and 60% on Claude-Sonnet-3.5 (24% improvement). This work highlights automated system design as a powerful paradigm for AI safety evaluation that can keep pace with rapidly evolving models.

💡 Research Summary

**
The paper introduces AgenticRed, an automated pipeline that uses large language models (LLMs) to design and evolve red‑team attack systems without any human‑written workflows. Traditional automated red‑team methods rely on manually crafted multi‑agent pipelines, which inherit human bias and limit exploration of the design space. AgenticRed reframes the problem: instead of optimizing a fixed attacker policy, it treats red‑team generation as a system‑design task. Building on Meta Agent Search, the authors represent an attack system as executable code that coordinates roles, tools, memory, and feedback loops. An initial “archive” stores several state‑of‑the‑art hand‑designed agents and their performance (attack success rate, ASR). A meta‑agent LLM iteratively generates new agents (“offspring”) from this archive, each of which is automatically executed against a target LLM and judged by a HarmBench‑style judge function. After evaluation on a malicious‑intent dataset, only the highest‑ASR agents are retained in the archive, imposing a Darwinian “survival of the fittest” pressure across generations.

Key technical contributions include: (1) a domain‑specific archive that incorporates execution interfaces for target‑model queries and judge feedback; (2) an evolutionary selection mechanism that creates multiple offspring per generation and keeps the best‑performing system; (3) helper functions that automate prompt generation, response retrieval, and success evaluation. The search space explores design patterns such as Dynamic Role‑Playing, Interactive Feedback Loops, and Monte‑Carlo Tree Search‑guided strategy selection.

Empirical evaluation covers open‑source models (Llama‑2‑7B, Llama‑3‑8B) and proprietary models (GPT‑3.5‑Turbo, GPT‑4o, Claude‑Sonnet‑3.5). On the HarmBench benchmark, AgenticRed achieves 96 % ASR on Llama‑2‑7B (a 36 % absolute gain over the previous best) and 98 % on Llama‑3‑8B. Transferability is striking: 100 % ASR on both GPT‑3.5‑Turbo and GPT‑4o, and a 60 % ASR on Claude‑Sonnet‑3.5 (24 % improvement). Qualitative analysis shows that the discovered workflows often combine dynamic role reassignment with iterative feedback, allowing the attacker to adapt prompts after partial failures and to exploit subtle guard‑rail gaps.

The authors acknowledge limitations: the malicious‑intent dataset is finite and may not capture all real‑world threats; the meta‑agent itself is a large LLM, incurring high computational cost; and the reliability of the judge function directly influences evolutionary pressure, potentially biasing the search. Future work is suggested in three directions: (a) extending the framework to multimodal red‑team tasks (e.g., image or audio generation); (b) integrating human‑in‑the‑loop verification or more robust safety judges to mitigate false positives; and (c) developing lightweight meta‑agents or more sample‑efficient evolutionary strategies to reduce compute.

In summary, AgenticRed demonstrates that automated system‑design techniques, when coupled with LLM in‑context learning and evolutionary selection, can automatically discover highly effective red‑team agents that outperform manually engineered baselines across a range of models. This establishes a scalable, adaptable approach for AI safety evaluation that can keep pace with the rapid rollout of increasingly capable language models.