用推理轨迹提升大模型对隐私政策的合规判断能力
Scaling Policy Compliance Assessment in Language Models with Policy Reasoning Traces
- 生成专门的推理链条作为桥梁，增强模型理解政策规则
- 在HIPAA和GDPR评估中达到新最优准确率
- 让模型更精准引用条款，适合合规审计场景使用
政策合规评估是判断输入内容是否严格遵守人为定义规则（即政策）的基础任务。实践中，人类专家会系统性地逐条分析政策条款以识别违规。然而，获取这种专家级推理过程的标注数据成本高昂。本文提出政策推理轨迹（Policy Reasoning Traces, PRT），一种专用生成的推理链，作为桥梁提升大语言模型的合规评估能力。实证表明，无论在推理时还是训练时使用PRT，均显著提升开源与商用模型的表现，在HIPAA和GDPR政策评估中达到新基准。此外，PRT还增强了模型准确引用政策条款的能力，并通过高利用率的原始思维链影响合规决策。
原文摘要 · Abstract (English)
Policy compliance assessment is a fundamental task of evaluating whether an input case strictly complies with a set of human-defined rules, more generally known as policies. In practice, human experts follow a systematic, step-by-step process to identify violations with respect to specific stipulations outlined in the policy. However, such documentation of gold-standard, expert-level reasoning processes is costly to acquire. In this paper, we introduce Policy Reasoning Traces (PRT), a form of specialized generated reasoning chains that serve as a reasoning bridge to improve an LLM's policy compliance assessment capabilities. Our empirical evaluations demonstrate that the use of PRTs for both inference-time and training-time scenarios significantly enhances the performance of open-weight and commercial models, setting a new state-of-the-art for HIPAA and GDPR policies. Beyond accuracy gains, we also highlight how PRTs can improve an LLM's ability to accurately cite policy clauses, as well as influence compliance decisions through their high utilization from the raw chains of thought.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。