A Causal Perspective for Enhancing Jailbreak Attack and Defense

Uncovering the mechanisms behind “jailbreaks” in large language models (LLMs) is crucial for enhancing their safety and reliability, yet these mechanisms remain poorly understood. Existing studies predominantly analyze jailbreak prompts by probing latent representations, often overlooking the causal relationships between interpretable prompt features and jailbreak occurrences. In this work, we propose Causal Analyst, a framework that integrates LLMs into data-driven causal discovery to identify the direct causes of jailbreaks and leverage them for both attack and defense. We introduce a comprehensive dataset comprising 35k jailbreak attempts across seven LLMs, systematically constructed from 100 attack templates and 50 harmful queries, annotated with 37 meticulously designed human-readable prompt features. By jointly training LLM-based prompt encoding and GNN-based causal graph learning, we reconstruct causal pathways linking prompt features to jailbreak responses. Our analysis reveals that specific features, such as “Positive Character” and “Number of Task Steps”, act as direct causal drivers of jailbreaks. We demonstrate the practical utility of these insights through two applications: (1) a Jailbreaking Enhancer that targets identified causal features to significantly boost attack success rates on public benchmarks, and (2) a Guardrail Advisor that utilizes the learned causal graph to extract true malicious intent from obfuscated queries. Extensive experiments, including baseline comparisons and causal structure validation, confirm the robustness of our causal analysis and its superiority over non-causal approaches. Our results suggest that analyzing jailbreak features from a causal perspective is an effective and interpretable approach for improving LLM reliability. Our code is available at https://github.com/Master-PLC/Causal-Analyst.

💡 Research Summary

The paper tackles the problem of jailbreaks in large language models (LLMs) from a causal perspective, aiming to uncover which human‑readable prompt attributes directly cause a model to produce harmful outputs. The authors first construct a large‑scale dataset comprising 35,000 jailbreak attempts across seven state‑of‑the‑art LLMs (including GPT‑4o, LLaMA 3, Baichuan 2, Qwen, etc.). Each attempt is generated by combining 100 template prompts (covering encryption, hijacking, and setting attack families) with 50 distinct harmful queries, resulting in 5,000 unique prompt‑query pairs that are evaluated on each model. Every prompt is annotated with 37 carefully designed features such as “Template Length,” “Number of Natural Languages,” “Positive Character,” “Command Tone,” “Number of Task Steps,” and various encryption‑related transformations. Feature annotation uses a hybrid pipeline: deterministic scripts for structural attributes and GPT‑4o for semantic or stylistic attributes, followed by manual verification.

To discover causal relationships, the authors propose the “Causal Analyst” framework, which integrates an LLM‑based encoder (Qwen2.5‑7B) with a graph neural network (GNN) that learns a directed acyclic graph (DAG) representing causal dependencies among features and the jailbreak outcome. The learning objective follows a continuous‑optimization approach similar to NOTEARS, allowing gradient‑based training while enforcing acyclicity. The model jointly optimizes the encoder and the DAG, using cross‑validation and expert feedback to validate edge directions. The resulting causal graph reveals that two features—“Positive Character” (i.e., framing the model as a benevolent entity) and “Number of Task Steps” (i.e., providing a multi‑step instruction chain)—have the strongest direct causal effect on the Answer Harmfulness (AH) response type. Other features such as specific encryption patterns, persona configuration, and temporal distortions act as indirect mediators.

The paper demonstrates two practical applications of the discovered causal knowledge. First, a “Jailbreaking Enhancer” modifies existing prompts by explicitly injecting the identified direct causal features. Experiments show that this enhancer raises attack success rates by an average of 18 percentage points over strong baselines, with the greatest gains observed when adding a positive persona and expanding the instruction chain to three‑to‑five steps. Second, a “Guardrail Advisor” leverages the learned causal graph to reverse‑engineer the true malicious intent behind obfuscated queries. By tracing back from observed features to latent intent, the advisor detects hidden jailbreak attempts with 92 % accuracy, outperforming conventional keyword‑based filters, especially on encrypted and setting‑based attacks.

Extensive evaluation compares the causal approach against non‑causal baselines (correlation analysis, feature importance from random forests, etc.). Across all seven LLMs, the causal model consistently yields higher attack success (for the enhancer) and higher detection precision/recall (for the advisor). Sensitivity analyses confirm that the key causal edges remain stable under different data splits and sampling variations, indicating robustness of the inferred structure.

The contributions are threefold: (1) introducing the first causal‑theoretic analysis of LLM jailbreaks with a publicly released 35 k‑sample dataset and 37‑feature annotation schema; (2) presenting a novel hybrid architecture that couples LLM embeddings with GNN‑based DAG learning to produce interpretable causal pathways; (3) showing that the identified causal factors can be directly exploited to both improve jailbreak attacks and strengthen defensive guardrails, achieving statistically significant performance gains. The authors argue that framing jailbreaks as causal mechanisms rather than mere correlations provides actionable insight for future safety research, and they suggest extensions to multimodal models, dynamic causal graph updates, and real‑time defense pipelines.