大模型代理会因隐藏信息违规,新框架通过模拟世界状态来提前识别风险。
Policy-Invisible Violations in LLM-Based Agents
- 构建了包含8类违规的基准测试,验证模型行为是否违反政策
- 引入哨兵系统,通过反事实图模拟预测动作后果并检查规则
- 在真实人类标注下准确率达93%,显著优于传统内容过滤方法
基于大模型的智能体可能执行语法正确、用户授权且语义合理的行为,但仍因决策时缺少组织政策判断所需的实体属性、上下文状态或会话历史而违反政策。我们称之为‘政策不可见违规’。为此,提出PhantomPolicy基准,涵盖八类违规场景,所有工具响应均仅含干净业务数据,无政策元数据。对五款前沿模型生成的600条推理轨迹进行人工复核,发现5.3%的原始标注需修正,凸显逐步追踪审查的重要性。进一步提出Sentinel框架,将每个动作视为对组织知识图谱的潜在修改,通过推测性执行生成动作后世界状态,并验证图结构不变性以决定允许、阻止或澄清。在人类标注的轨迹上,Sentinel准确率达93.0%,远超仅依赖内容的DLP基线(68.8%),精度保持较高,但特定类别仍有提升空间。结果表明,一旦政策相关世界状态可被获取,合规控制即可实现质的飞跃。
原文摘要 · Abstract (English)
LLM-based agents can execute actions that are syntactically valid, user-sanctioned, and semantically appropriate, yet still violate organizational policy because the facts needed for correct policy judgment are hidden at decision time. We call this failure mode policy-invisible violations: cases in which compliance depends on entity attributes, contextual state, or session history absent from the agent's visible context. We present PhantomPolicy, a benchmark spanning eight violation categories with balanced violation and safe-control cases, in which all tool responses contain clean business data without policy metadata. We manually review all 600 model traces produced by five frontier models and evaluate them using human-reviewed trace labels. Manual review changes 32 labels (5.3%) relative to the original case-level annotations, confirming the need for trace-level human review. To demonstrate what world-state-grounded enforcement can achieve under favorable conditions, we introduce Sentinel, an enforcement framework based on counterfactual graph simulation. Sentinel treats every agent action as a proposed mutation to an organizational knowledge graph, performs speculative execution to materialize the post-action world state, and verifies graph-structural invariants to decide Allow/Block/Clarify. Against human-reviewed trace labels, Sentinel substantially outperforms a content-only DLP baseline (68.8% vs. 93.0% accuracy) while maintaining high precision, though it still leaves room for improvement on certain violation categories. These results demonstrate what becomes achievable once policy-relevant world state is made available to the enforcement layer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。