정책을 자동 규칙으로 전환하는 P2T 프레임워크 AI 가이드라인의 실행 가능성 향상

📝 Abstract

AI policy guidance is predominantly written as prose, which practitioners must first convert into executable rules before frameworks can evaluate or enforce them. This manual step is slow, error-prone, difficult to scale, and often delays the use of safeguards in real-world deployments. To address this gap, we present Policy→Tests (P2T), a framework that converts natural-language policy documents into normalized, machine-readable rules. The framework comprises a pipeline and a compact domain-specific language (DSL) that encodes hazards, scope, conditions, exceptions, and required evidence, yielding a canonical representation of extracted rules. To test the framework beyond a single policy, we apply it across general frameworks, sector guidance, and enterprise standards, extracting obligation-bearing clauses and converting them into executable rules. These AI-generated rules closely match strong human baselines on span-and rulelevel metrics, with robust inter-annotator agreement on the gold set. To evaluate downstream behavioral and safety impact, we add HIPAA-derived safeguards to a generative agent and compare it with an otherwise identical agent without guardrails. An LLM-based judge, aligned with gold-standard criteria, measures violation rates and robustness to obfuscated and compositional prompts. Detailed results are provided in the appendix. We release the codebase, DSL, prompts, and rule sets as open-source resources to enable reproducible evaluation.

💡 Analysis

AI policy guidance is predominantly written as prose, which practitioners must first convert into executable rules before frameworks can evaluate or enforce them. This manual step is slow, error-prone, difficult to scale, and often delays the use of safeguards in real-world deployments. To address this gap, we present Policy→Tests (P2T), a framework that converts natural-language policy documents into normalized, machine-readable rules. The framework comprises a pipeline and a compact domain-specific language (DSL) that encodes hazards, scope, conditions, exceptions, and required evidence, yielding a canonical representation of extracted rules. To test the framework beyond a single policy, we apply it across general frameworks, sector guidance, and enterprise standards, extracting obligation-bearing clauses and converting them into executable rules. These AI-generated rules closely match strong human baselines on span-and rulelevel metrics, with robust inter-annotator agreement on the gold set. To evaluate downstream behavioral and safety impact, we add HIPAA-derived safeguards to a generative agent and compare it with an otherwise identical agent without guardrails. An LLM-based judge, aligned with gold-standard criteria, measures violation rates and robustness to obfuscated and compositional prompts. Detailed results are provided in the appendix. We release the codebase, DSL, prompts, and rule sets as open-source resources to enable reproducible evaluation.

📄 Content

Executable Governance for AI: Translating Policies into Rules Using LLMs Gautam Varma Datla1, Anudeep Vurity2, Tejaswani Dash3, Tazeem Ahmad4, Mohd Adnan5, Saima Rafi6 1New Jersey Institute of Technology, Newark, USA 2,3George Mason University, Fairfax, USA 4School of Mathematics, Physics and Computing, University of Southern Queensland, Australia 5University of Aveiro, Aveiro, Portugal 6Edinburgh Napier University, Edinburgh, Scotland, UK 1gvd6@njit.edu, 2avurity@gmu.edu, 3tdash@gmu.edu, 4Tazeem.Ahmad@unisq.edu.au, 5m.adnan1821@gmail.com, 6s.rafi@napier.ac.uk Abstract AI policy guidance is predominantly written as prose, which practitioners must first convert into executable rules before frameworks can evaluate or enforce them. This manual step is slow, error-prone, difficult to scale, and often delays the use of safeguards in real-world deployments. To address this gap, we present Policy→Tests (P2T), a framework that converts natural-language policy documents into normalized, machine-readable rules. The framework comprises a pipeline and a compact domain-specific language (DSL) that en- codes hazards, scope, conditions, exceptions, and required evidence, yielding a canonical representation of extracted rules. To test the framework beyond a single policy, we ap- ply it across general frameworks, sector guidance, and en- terprise standards, extracting obligation-bearing clauses and converting them into executable rules. These AI-generated rules closely match strong human baselines on span- and rule- level metrics, with robust inter-annotator agreement on the gold set. To evaluate downstream behavioral and safety im- pact, we add HIPAA-derived safeguards to a generative agent and compare it with an otherwise identical agent without guardrails. An LLM-based judge, aligned with gold-standard criteria, measures violation rates and robustness to obfuscated and compositional prompts. Detailed results are provided in the appendix. We release the codebase, DSL, prompts, and rule sets as open-source resources to enable reproducible evaluation. Code — https://github.com/gautamvarmadatla/Policy- Tests-P2T-for-operationalizing-AI-governance Introduction As artificial intelligence (AI) permeates critical sectors, en- suring its responsible use has become imperative (Cheng et al. 2025; Shen et al. 2024; Carlini et al. 2021; Wei, Shi et al. 2024). In response, governments and industry bod- ies have introduced governance frameworks. For example, the European Union AI Act sets obligations for high risk uses (European Union 2024), and the NIST AI Risk Man- agement Framework provides a voluntary guide focused on Copyright © 2026, Association for the Advancement of Artificial Intelligence (www.aaai.org ). All rights reserved. trustworthiness (NIST 2023). Building on these efforts, ad- ditional standards and principles promote responsible prac- tice, including the OECD AI Principles (Organisation for Economic Co-operation and Development 2019) and ISO IEC 42001 for AI management systems (International Orga- nization for Standardization and International Electrotech- nical Commission 2023). However, these instruments are intentionally nonprescriptive. The AI RMF explicitly states that it is intended to be voluntary, rights preserving, and use case agnostic, and the companion Playbook offers sugges- tions that organizations may adopt as needed, rather than mandated tests (National Institute of Standards and Tech- nology 2023). Efforts like Singapore’s AI Verify show how process checks and technical evaluations can be packaged into a single toolkit, yet teams still spend expert hours turn- ing broad guidance into the concrete checks that a specific system will pass or fail in a repeatable way (Commission 2021). From here, the gap unfolds in practice, as organiza- tions must invent concrete, verifiable procedures to demon- strate compliance in deployed systems, and this shortfall can erode public trust in high-stakes settings. Our work targets this gap by asking whether policy text can be turned into checks that fit naturally into an engineer- ing workflow. We first ask whether an automated pipeline can extract policy rules with span level and field level qual- ity that stands up to careful human work. We then ask whether rules derived from policy actually reduce observed violations when they are used to evaluate the behavior of AI agents under a consistent runtime. Finally, we examine which controls matter most for the quality and robustness of extracted rules, including schema guarded decoding, tar- geted repairs, evidence gating, semantic deduplication, SMT conflict checks, and counterfactual flips. To address these questions we present a Policy→Tests (P2T) framework that converts policy documents into exe- cutable rules without breaking stride. The pipeline ingests policy documents and finds spans that are likely to contain obligations or prohibitions, then performs schema guarded LLM extraction supported by deterministic checks

This content is AI-processed based on ArXiv data.