Auto-Tuning Safety Guardrails for Black-Box Large Language Models

Large language models (LLMs) are increasingly deployed behind safety guardrails such as system prompts and content filters, especially in settings where product teams cannot modify model weights. In practice these guardrails are typically hand-tuned, brittle, and difficult to reproduce. This paper studies a simple but practical alternative: treat safety guardrail design itself as a hyperparameter optimization problem over a frozen base model. Concretely, I wrap Mistral-7B-Instruct with modular jailbreak and malware system prompts plus a ModernBERT-based harmfulness classifier, then evaluate candidate configurations on three public benchmarks covering malware generation, classic jailbreak prompts, and benign user queries. Each configuration is scored using malware and jailbreak attack success rate, benign harmful-response rate, and end-to-end latency. A 48-point grid search over prompt combinations and filter modes establishes a baseline. I then run a black-box Optuna study over the same space and show that it reliably rediscovers the best grid configurations while requiring an order of magnitude fewer evaluations and roughly 8x less wall-clock time. The results suggest that viewing safety guardrails as tunable hyperparameters is a feasible way to harden black-box LLM deployments under compute and time constraints.

💡 Research Summary

The paper tackles a practical problem that many product teams face when deploying large language models (LLMs) as managed services: the underlying model weights cannot be altered, yet safety must be enforced through deployment‑time guardrails such as system prompts and content filters. Rather than hand‑tuning these guardrails, the author treats the selection of prompts and filter policies as a discrete hyperparameter optimization problem around a frozen base model.

Methodology

• Base model: Mistral‑7B‑Instruct‑v0.2, run on an A100 GPU via HuggingFace Transformers.
• Safety components: Four binary “snippets” that can be toggled on or off (JB1, JB2 for jailbreak deterrence; MW1, MW2 for malware deterrence) and three filter modes (none, mild with a 0.5 harmfulness threshold, strict with a 0.8 threshold).
• Harmfulness classifier: ModernBERT‑wildguardmix, a binary classifier trained on safety‑related text, used both to flag responses and to replace them with canned refusals when the threshold is exceeded.
• Configuration space: 2⁴ × 3 = 48 distinct guardrail configurations.

Benchmarks
Three publicly available datasets are used, each sampled to 50 prompts:

1. RMCBench – malicious code generation prompts (malware).
2. ChatGPT‑Jailbreak‑Prompts – classic jailbreak attempts.
3. JBB‑Behaviors – benign user queries.

Metrics
Four quantitative metrics are computed per configuration:

• Malware attack success rate (ASRₘₐₗ).
• Jailbreak attack success rate (ASRⱼᵦ).
• Benign harmful‑response rate (a proxy for over‑refusal or hallucination).
• Average latency (generation + filtering time).

A scalar objective for optimization is defined as
score = 0.4·ASRₘₐₗ + 0.4·ASRⱼᵦ + 0.1·HarmBen + 0.1·Latency
with lower scores indicating better safety‑usability trade‑offs.

Experiments

1. Baseline (grid search) – All 48 configurations are evaluated on the full 50‑prompt sets for each benchmark, establishing a performance frontier. The bare model (no prompts, filter‑none) exhibits extremely high vulnerability: malware ASR ≈ 0.48, jailbreak ASR ≈ 0.98, and benign harmful‑response ≈ 0.42.
2. Effect of filtering alone – Adding only the classifier‑based filter reduces malware ASR by ~10 pp (to ≈ 0.38) with modest latency overhead, but jailbreak ASR remains high.
3. Prompt + filter combos – Configurations that enable both jailbreak and malware snippets together with mild filtering achieve the best balance: malware ASR drops further, jailbreak ASR improves modestly, and benign harmful‑response falls to ≈ 0.22. This demonstrates complementary benefits of pre‑emptive prompt reminders and post‑generation filtering.
4. Black‑box hyperparameter optimization – Optuna is used to explore the same space. The search space mirrors the grid (four binary variables + one categorical filter mode). A fast loop evaluates each trial on only 10 prompts per dataset (24 trials total). The top‑5 trials are then re‑evaluated on the full 50‑prompt sets. Optuna converges on configurations that match or slightly improve the best grid results, while requiring roughly one‑tenth the number of model evaluations and about eight times less wall‑clock time. Pareto plots show Optuna quickly homes in on the safety‑latency frontier, confirming that standard black‑box optimizers can efficiently locate high‑performing guardrail settings.

Discussion & Limitations

• Data scope: Only 50 examples per benchmark and English‑only prompts limit statistical confidence and generalizability.
• Classifier bias: The same ModernBERT classifier is used for both evaluation and enforcement, potentially masking systematic errors. Separate human‑in‑the‑loop or orthogonal classifiers would yield more reliable safety estimates.
• Single‑turn focus: Multi‑turn jailbreak or persuasion attacks are not covered; extending to dialogue‑level attacks is a natural next step.
• Scalar objective: The weighting (0.4, 0.4, 0.1, 0.1) is hand‑picked; real deployments may require constrained or multi‑objective formulations that treat malware, jailbreak, and user‑experience as distinct priorities.
• Configuration richness: The explored space (four binary prompts, three filter modes) is modest. Real‑world systems may need dynamic policies, per‑domain thresholds, routing to secondary models, or adaptive refusal strategies. The hyperparameter‑optimization framing, however, is readily extensible to richer spaces.

Related Work
The paper situates itself at the intersection of prompt‑based defenses, content‑filter classifiers, and black‑box hyperparameter optimization (Bayesian, evolutionary). Prior work has shown that self‑reminder prompts can mitigate some jailbreaks and that safety classifiers catch many unsafe generations, but few studies have framed guardrail design itself as an optimization problem.

Conclusion
The study demonstrates that treating safety guardrails as tunable hyperparameters is both feasible and beneficial. Using a modest experimental setup (Mistral‑7B‑Instruct, ModernBERT classifier, three public benchmarks), the author shows that: (1) without guardrails the model is highly vulnerable; (2) simple combinations of system prompts and classifier‑based filtering substantially improve safety with modest latency cost; (3) off‑the‑shelf black‑box optimizers like Optuna can discover near‑optimal guardrail configurations an order of magnitude faster than exhaustive grid search. While the scale is limited, the methodology is directly applicable to product teams that already perform hyperparameter tuning for model training. Future work should expand the guardrail search space, incorporate richer safety benchmarks (hate speech, self‑harm, data leakage), evaluate multi‑turn attacks, and integrate human evaluation to move toward deployable tooling for systematic hardening of black‑box LLM applications.