Capability-Based Scaling Trends for LLM-Based Red-Teaming

As large language models grow in capability and agency, identifying vulnerabilities through red-teaming becomes vital for safe deployment. However, traditional prompt-engineering approaches may prove ineffective once red-teaming turns into a \emph{weak-to-strong} problem, where target models surpass red-teamers in capabilities. To study this shift, we frame red-teaming through the lens of the \emph{capability gap} between attacker and target. We evaluate more than 600 attacker-target pairs using LLM-based jailbreak attacks that mimic human red-teamers across diverse families, sizes, and capability levels. Three strong trends emerge: (i) more capable models are better attackers, (ii) attack success drops sharply once the target’s capability exceeds the attacker’s, and (iii) attack success rates correlate with high performance on social science splits of the MMLU-Pro benchmark. From these observations, we derive a \emph{jailbreaking scaling curve} that predicts attack success for a fixed target based on attacker-target capability gap. These findings suggest that fixed-capability attackers (e.g., humans) may become ineffective against future models, increasingly capable open-source models amplify risks for existing systems, and model providers must accurately measure and control models’ persuasive and manipulative abilities to limit their effectiveness as attackers.

💡 Research Summary

The paper “Capability‑Based Scaling Trends for LLM‑Based Red‑Teaming” investigates how the relative capabilities of attacker and target language models affect the success of jailbreak attacks, a key component of safety testing for large language models (LLMs). The authors introduce the notion of a “capability gap” – the difference between an attacker’s and a target’s performance on the MMLU‑Pro benchmark – and systematically explore its impact on attack success rates (ASR).

To do this, they construct a large experimental matrix of over 600 attacker‑target pairs, covering a wide spectrum of model families (Llama 2, Llama 3, Mistral, Vicuna, Qwen 2.5) and sizes, as well as several closed‑source systems (Gemini, GPT‑4.1, Claude‑3.7). Four representative human‑like jailbreak methods are employed: PAIR, TAP, PAP (all single‑turn) and Crescendo (multi‑turn). Each attack is run up to 25 inner steps on the HarmBench benchmark, with a neutral judge evaluating whether the target’s response violates safety constraints.

Three robust trends emerge. First, an attacker’s average ASR scales almost linearly with its overall MMLU‑Pro score (Spearman ρ > 0.88). More capable models generate more sophisticated prompts, better role‑play, and more persuasive language, making them stronger red‑teamers. Second, for any fixed target, ASR follows a sigmoid curve as a function of the capability gap: when the attacker’s score exceeds the target’s by a modest margin, success approaches 100 %; when the target is even slightly more capable, success drops sharply toward 0 %. This “capability‑gap scaling curve” provides a predictive tool for future safety assessments. Third, performance on the social‑science subset of MMLU‑Pro (health, psychology, philosophy, etc.) correlates more strongly with ASR than STEM performance, indicating that persuasive, manipulative, and ethical reasoning abilities are the primary drivers of jailbreak success.

The authors discuss several implications. Human red‑teamers, whose effective capability is roughly bounded (≈150 MMLU‑Pro), will become increasingly ineffective as open‑source models surpass this threshold, turning red‑teaming into a weak‑to‑strong problem. Conversely, as open‑source models become stronger attackers, they pose heightened risk to existing deployed systems. Therefore, model providers should benchmark and constrain models’ “persuasion” and “manipulation” abilities separately from standard accuracy metrics, and incorporate safeguards that limit these capabilities.

Overall, the study reframes LLM safety evaluation from a pure performance‑centric view to one that explicitly accounts for the relative power of attacker and defender. By quantifying the scaling behavior of jailbreak success with capability gaps, it offers a practical framework for forecasting future red‑team effectiveness, guiding the design of automated, large‑scale red‑team pipelines, and highlighting the need for new safety metrics focused on social‑cognitive skills.