arXiv:2607.07405cs.AIcs.CR2026-07被引 2

给大模型工具调用加检查关卡,能发现并阻止隐性违规操作。

Reason Less, Verify More: Deterministic Gates Recover a Silent Policy-Violation Failure Mode in Tool-Using LLM Agents

  • 用只读预执行门控检查动作前的状态与调用请求
  • 使任务成功率从29.6%提升至42.0%,提升12.4个百分点
  • 特别适合高风险领域中需严格遵守规则的智能体系统

使用工具的大语言模型智能体可能在看似成功完成任务的同时违反其应遵守的策略。在策略宽松环境中,即使状态转移被领域策略禁止,工具仍可执行任何格式正确的调用,导致无声错误状态(如取消预订、更改乘客数、未验证即处理索赔),且工具和智能体自报告均无法暴露此类问题。我们在τ²-bench航空领域研究此故障模式:在预算智能体上,78%的观测失败为无工具错误的无声状态失败,整体失败率在不同种子下可重复,非采样噪声所致。随后评估一种轻量干预:在执行前通过确定性只读门控检查提议调用与当前状态。四门控组合使gpt-4o-mini在全基准上的成功率从29.6%提升至42.0%(+12.4pp;配对任务级自助法P=0.0012),该提升在独立15种子集上重现(+12.3pp;P=0.0008)。效果集中于门控触发的任务:在26/50个触发任务中,成功率提升19.2pp,而24个未触发任务的变动不排斥零。两个负向对照(自约束零售域与BFCL)限定机制适用范围:仅在工具策略宽松时有效,工具已自约束时帮助甚微。作为辅助证据,前沿模型gpt-5.2默认推理下仍尝试政策违规写入,相同门控组合使其成功率从61.2%提升至71.6%(+10.4pp;P=0.020;n=5,无复现)。贡献在于提供一个有界评估与可靠性结果:确定性门控虽不能保证任务成功,但可在动作边界上确定性地防止已知类别的沉默政策违规写入。

原文摘要 · Abstract (English)

Tool-using LLM agents can violate the very policies they are deployed to enforce while appearing to complete the task successfully. In policy-permissive environments, a tool may execute any well-formed call even when the corresponding state transition is forbidden by domain policy. The result is a silent wrong state (a booking cancelled, a passenger count changed, a claim acted on without verification) that neither the tool nor the agent's self-report exposes. We study this failure mode in the $τ^2$-bench airline domain. On a budget agent, 78% of observed failures are silent wrong-state failures with no tool error, and the aggregate failure rate is reproducible across disjoint seeds, not sampling noise. We then evaluate a lightweight intervention: deterministic, read-only pre-execution gates that inspect the proposed call and current state before allowing a write. A four-gate suite raises full-benchmark success from 29.6% to 42.0% on gpt-4o-mini (+12.4pp; paired task-level bootstrap P=0.0012), and the lift reproduces on a disjoint 15-seed set (+12.3pp; P=0.0008). The effect is concentrated where the gates fire: on the 26/50 firing tasks, success rises by +19.2pp, while movement on the 24 non-firing tasks does not exclude zero. Two negative controls (a self-enforcing retail domain and BFCL) bound the mechanism: gates help when tools are policy-permissive and add little where tools already self-enforce. As suggestive evidence, not a central claim, the same failure mode persists at the frontier: gpt-5.2 at default reasoning still attempts policy-violating writes, and the same suite improves success from 61.2% to 71.6% (+10.4pp; P=0.020; n=5, no replication). The contribution is a bounded evaluation and reliability result: deterministic gates do not guarantee task success, but they can deterministically prevent a known class of silent policy-violating writes at the action boundary.

大模型安全智能体系统策略合规门控机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。