用大模型自动把政策文本转成可执行规则,提升AI安全落地效率
Executable Governance for AI: Translating Policies into Rules Using LLMs
- 构建P2T框架,将政策文字段落转化为标准化机器规则
- 生成规则在跨度和规则级别上接近人工基准,一致性高
- 适用于医疗等领域的安全防护,适合AI治理研究者使用
AI政策多以自然语言描述,从业者需手动转化为可执行规则后才能评估或执行,该过程耗时、易错且难扩展,常导致安全措施延迟部署。为此,我们提出Policy-to-Tests(P2T)框架,将自然语言政策文档转换为标准化、机器可读的规则。框架包含一个处理流程和轻量级领域特定语言(DSL),用于编码风险、范围、条件、例外与证据要求,形成规则的规范表示。为验证框架泛化能力,我们在通用框架、行业指南及企业标准中应用该方法,提取具有义务性的条款并转化为可执行规则。生成规则在跨度级与规则级指标上接近强基线人类标注结果,金标集上表现出稳健的标注者间一致性。为评估下游行为与安全影响,我们将基于HIPAA的防护措施加入生成式代理,并与无防护版本对比。使用对齐于黄金标准的LLM裁判评估违规率及对抗性提示下的鲁棒性。详细结果见附录。代码库、DSL、提示词与规则集已开源,支持可复现评估。
原文摘要 · Abstract (English)
AI policy guidance is predominantly written as prose, which practitioners must first convert into executable rules before frameworks can evaluate or enforce them. This manual step is slow, error-prone, difficult to scale, and often delays the use of safeguards in real-world deployments. To address this gap, we present Policy-to-Tests (P2T), a framework that converts natural-language policy documents into normalized, machine-readable rules. The framework comprises a pipeline and a compact domain-specific language (DSL) that encodes hazards, scope, conditions, exceptions, and required evidence, yielding a canonical representation of extracted rules. To test the framework beyond a single policy, we apply it across general frameworks, sector guidance, and enterprise standards, extracting obligation-bearing clauses and converting them into executable rules. These AI-generated rules closely match strong human baselines on span-level and rule-level metrics, with robust inter-annotator agreement on the gold set. To evaluate downstream behavioral and safety impact, we add HIPAA-derived safeguards to a generative agent and compare it with an otherwise identical agent without guardrails. An LLM-based judge, aligned with gold-standard criteria, measures violation rates and robustness to obfuscated and compositional prompts. Detailed results are provided in the appendix. We release the codebase, DSL, prompts, and rule sets as open-source resources to enable reproducible evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。