发现智能体流程中隐蔽的政策规避问题,揭示现有评估漏洞
Near-Miss: Latent Policy Failure Detection in Agentic Workflows
- 通过分析工具调用决策是否充分知情,检测政策绕过风险
- 8%-17%的正确最终结果轨迹存在潜在违规行为
- 适合关注AI合规性与流程安全的研究者和开发者
基于大模型的智能体系统在业务自动化中需遵守状态更新的政策规则。当前评估通常仅对比最终系统状态与预期结果,虽能发现明显违规,却可能忽略更隐蔽的问题:智能体绕过必要政策检查,但因偶然情况达成正确结果。这类情形称为近似失误或潜在故障。本文提出一种新指标,用于检测代理对话轨迹中的潜在政策失败。基于ToolGuard框架将自然语言政策转为可执行保护代码,我们的方法分析代理轨迹,判断其工具调用决策是否充分知情。在τ²-验证的Airlines基准上,对多个主流开源及专有大模型进行测试,结果显示:即使最终结果符合预期,涉及状态修改工具调用的轨迹中仍有8%-17%存在潜在故障。这揭示了当前评估方法的盲点,强调需同时考察决策过程而非仅关注最终结果。
原文摘要 · Abstract (English)
Agentic systems for business process automation often require compliance with policies governing conditional updates to the system state. Evaluation of policy adherence in LLM-based agentic workflows is typically performed by comparing the final system state against a predefined ground truth. While this approach detects explicit policy violations, it may overlook a more subtle class of issues in which agents bypass required policy checks, yet reach a correct outcome due to favorable circumstances. We refer to such cases as near-misses or latent failures. In this work, we introduce a novel metric for detecting latent policy failures in agent conversations traces. Building on the ToolGuard framework, which converts natural-language policies into executable guard code, our method analyzes agent trajectories to determine whether agent's tool-calling decisions where sufficiently informed. We evaluate our approach on the $τ^2$-verified Airlines benchmark across several contemporary open and proprietary LLMs acting as agents. Our results show that latent failures occur in 8-17% of trajectories involving mutating tool calls, even when the final outcome matches the expected ground-truth state. These findings reveal a blind spot in current evaluation methodologies and highlight the need for metrics that assess not only final outcomes but also the decision process leading to them.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。