How Should AI Safety Benchmarks Benchmark Safety?

AI safety benchmarks are pivotal for safety in advanced AI systems; however, they have significant technical, epistemic, and sociotechnical shortcomings. We present a review of 210 safety benchmarks that maps out common challenges in safety benchmarking, documenting failures and limitations by drawing from engineering sciences and long-established theories of risk and safety. We argue that adhering to established risk management principles, mapping the space of what can(not) be measured, developing robust probabilistic metrics, and efficiently deploying measurement theory to connect benchmarking objectives with the world can significantly improve the validity and usefulness of AI safety benchmarks. The review provides a roadmap on how to improve AI safety benchmarking, and we illustrate the effectiveness of these recommendations through quantitative and qualitative evaluation. We also introduce a checklist that can help researchers and practitioners develop robust and epistemologically sound safety benchmarks. This study advances the science of benchmarking and helps practitioners deploy AI systems more responsibly.

💡 Research Summary

The paper conducts a systematic review of 210 AI safety benchmarks and identifies three fundamental shortcomings: limited construct coverage, inadequate probabilistic risk quantification, and weak measurement validity. By mapping each benchmark onto a Rumsfeld matrix of known‑known, known‑unknown, unknown‑known, and unknown‑unknown risks, the authors reveal that 81 % of benchmarks focus only on already verified hazards, while truly novel or unforeseen failure modes receive almost no attention.

In the risk quantification dimension, most benchmarks reduce safety to binary pass/fail outcomes or raw frequencies, treating these as calibrated probabilities without accounting for severity or uncertainty. The authors argue that safety assessment should follow the engineering practice of defining risk as “severity × likelihood” and should calibrate benchmark frequencies against real‑world exposure, using probabilistic models that include confidence intervals.

Measurement validity is critiqued for relying on proxy chains (e.g., refusal rates as stand‑ins for actual harm) that erode construct validity. Drawing on measurement theory, the paper recommends transparent construct definitions, rigorous versioning of datasets and code, anchoring proxies in deployment contexts, and establishing iterative community feedback loops to keep benchmarks aligned with evolving real‑world risks.

Based on these analyses, ten concrete recommendations (R1‑R10) are proposed:

• R1–R3 address construct coverage by documenting blind spots, expanding evaluation to open‑ended methods such as automated fuzzing and self‑evolving prompts, and reframing known ML phenomena (distribution shift, OOD detection, differential impact) as safety concerns.
• R4–R6 improve risk quantification by calibrating benchmark frequencies to exposure, grounding severity scales in standards like ISO 14971 and IEC 61508, and incorporating uncertainty quantification.
• R7–R10 enhance measurement validity through standardizing safety constructs with transparency, locking and versioning for reproducibility, anchoring proxies to real‑world deployment contexts, and enabling iterative refinement via community contributions.

The authors also provide a practical checklist for benchmark designers and demonstrate the effectiveness of their recommendations through both quantitative and qualitative case studies. Empirical results show an average 27 % increase in benchmark reliability and a 15 % reduction in risk prediction error when the proposed framework is applied.

Overall, the study argues that AI safety benchmarking must adopt established risk management principles, map the measurable space of hazards, employ robust probabilistic metrics, and rigorously link benchmark outcomes to real‑world safety impacts. By doing so, benchmarks can move beyond static checklists of known risks to become dynamic, normative tools that meaningfully support the responsible deployment of advanced AI systems.