检测工具调用隐性失败，自动触发恢复操作提升任务成功率
Outcome Monitors: Recovery Affordances for Silent Tool Failures
- 通过分析任务无关轨迹挖掘结果契约，实时检测工具输出异常
- 在多模型测试中任务完成率从10.9%提升至28.1%，零售场景提升12-14点
- 适用于高可靠性要求的智能体系统，尤其适合存在隐性故障场景
当工具调用超时时，智能体可感知失败并绕行。但缓存错误页面或负价格可能以预期格式返回并被误认为事实。本文提出Outcome Monitors，通过从任务无关轨迹中挖掘或基于公开模式推导结果契约，检测契约违反情况。一旦违反，监控器保留结果并发出非绑定收据，标明违反属性及公开恢复工具。在冻结预设评估中注入故障，Outcome Monitors使ToolMaze任务完成率在四个模型、两组供应商中从10.9%提升至28.1%，并在第三组复现。在tau-bench零售测试中，两个层级分别提升14.0和12.0分。独立控制实验表明，移除恢复工具列表则收益消失，恢复后效果重现；诊断细节与时间无显著影响。增益集中于故障阻塞任务完成的情况。在基于公开故障分类体系的套件测试中，超出挖掘词表的检测率降至46%，但交付继续且完成率不变。恢复工具是这些控制中的活跃收据内容；将检测扩展至契约词表外仍为开放问题。
原文摘要 · Abstract (English)
When a tool call times out, the agent sees the failure and can route around it. A cached error page or negative price can instead arrive in the expected format and be consumed as fact. We introduce Outcome Monitors, which detect violations of outcome contracts mined from task-disjoint traces or derived from public schemas. On a violation, the monitor preserves the result and issues a nonbinding receipt naming the violated property and public recovery tools. In frozen, prespecified evaluations with injected failures, Outcome Monitors raise ToolMaze completion from 10.9% to 28.1% across four models in two provider families and replicate in a third. In tau-bench retail, completion improves by 14.0 and 12.0 points on two tiers. In separate ToolMaze controls, removing the recovery-tool list eliminates the measured gain and restoring it recovers the effect; diagnostic detail and timing produce no detectable differences. Gains concentrate where the fault blocks completion. On a suite transcribed from a published incident taxonomy, detection outside the mined vocabulary falls to 46%, though delivery continues and completion is unchanged. Recovery tools are the active receipt content in these controls; extending detection beyond the contract vocabulary remains open.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。