Frontier AI Auditing: Toward Rigorous Third-Party Assessment of Safety and Security Practices at Leading AI Companies

We outline a vision for frontier AI auditing, which we define as rigorous third-party verification of frontier AI developers’ safety and security claims, and evaluation of their systems and practices against relevant standards, based on deep, secure access to non-public information. Frontier AI audits should not be limited to a company’s publicly deployed products, but should instead consider the full range of organization-level safety and security risks, including internal deployment of AI systems, information security practices, and safety decision-making processes. We describe four AI Assurance Levels (AALs), the higher levels of which provide greater confidence in audit findings. We recommend AAL-1 as a baseline for frontier AI generally, and AAL-2 as a near-term goal for the most advanced subset of frontier AI developers. Achieving the vision we outline will require (1) ensuring high quality standards for frontier AI auditing, so it does not devolve into a checkbox exercise or lag behind changes in the industry; (2) growing the ecosystem of audit providers at a rapid pace without compromising quality; (3) accelerating adoption of frontier AI auditing by clarifying and strengthening incentives; and (4) achieving technical readiness for high AI Assurance Levels so they can be applied when needed.

💡 Research Summary

The paper proposes a comprehensive framework for “frontier AI auditing,” defined as rigorous third‑party verification of safety and security claims made by developers of the most advanced AI systems. Recognizing that frontier AI—general‑purpose models whose performance is within a year of the state‑of‑the‑art—poses unprecedented societal risks, the authors argue that existing transparency‑only approaches are insufficient. They advocate for independent auditors who obtain deep, secure access to non‑public information and evaluate organizations holistically rather than focusing solely on individual products.
The authors delineate four risk categories that audits must cover: (1) intentional misuse (e.g., weaponization, cyber‑attacks), (2) unintended harmful behavior (e.g., erroneous medical advice, buggy code), (3) information‑security threats (theft of models or data), and (4) emergent social harms (addiction, self‑harm facilitation). For each category, auditors should both verify corporate claims and benchmark practices against regulations, industry standards, and best‑practice guidelines.
Central to the proposal is the AI Assurance Level (AAL) taxonomy, comprising four escalating levels of confidence. AAL‑1 corresponds to the current best practice: a time‑bounded, API‑based assessment lasting a few weeks, with limited non‑public data. AAL‑2 expands scope by granting auditors broader access to internal documentation, training pipelines, compute allocation, governance records, and staff interviews, enabling a more holistic organizational risk assessment. AAL‑3 and AAL‑4 envision continuous oversight with white‑box access, automated monitoring, and the ability to detect active deception attempts; however, the paper acknowledges that technical and organizational readiness for these highest levels is not yet achieved and outlines a research agenda to bridge the gap.
To balance deep access with protection of intellectual property and sensitive data, the authors recommend mechanisms drawn from other sectors: on‑site NDA‑bound data rooms, selective sharing with a vetted subset of auditors, and AI‑driven summarization or analysis of highly confidential material. Independence safeguards include mandatory disclosure of financial relationships, standardized engagement contracts that prevent “shopping” for favorable auditors, and cooling‑off periods for personnel moving between industry and audit roles. Alternative payment models that reduce dependence on the auditee are also suggested.
Methodologically, audits should follow a standardized process while allowing auditors discretion to tailor metrics, employ automated tools, and adapt scope as issues emerge. Results must be communicated clearly through structured reports that specify scope, assurance level, findings, reasoning, and recommendations. To protect proprietary information, auditors can issue redacted public summaries while providing full reports to boards, executives, and regulators.
The paper identifies four critical challenges and corresponding next steps: (1) establishing high‑quality standards to prevent audits from devolving into checklist exercises; (2) rapidly scaling the ecosystem of audit providers without sacrificing rigor, possibly via a tiered accreditation program; (3) creating strong incentives for adoption by investors, insurers, and policymakers (e.g., insurance premium discounts, procurement requirements for high‑risk systems); and (4) achieving technical readiness for AAL‑3 and AAL‑4 through targeted R&D funding, pilot projects, and public‑private collaborations.
Overall, the authors argue that a robust frontier AI auditing regime will enhance safety and security outcomes, provide a feedback loop for evolving standards, and enable more confident deployment of powerful AI in high‑stakes domains such as health and defense. By aligning standards, ecosystem development, incentives, and technical capability, the proposed framework seeks to build the trust infrastructure necessary for the responsible advancement of frontier AI.