arXiv:2608.26189cs.AIcs.CL2026-08

揭示工具调用智能体失败根源,提出新评估方法提升可靠性

Invocation-Level Reliability of Tool-Using Agents

  • 区分工具选择与推理错误,分层评估调用正确率
  • 深度6时自运行下准确率仅剩68.4%,早期错误严重拖累后续
  • 提出状态条件评分法,可零成本修复评估偏差

工具使用智能体的失败源于错误选择工具或形成错误推理,早期任一失败都会无声地破坏后续结果。本文在无污染的多步任务(深度1-8)中,对五种开源模型在教师强制和自主运行两种情境下,分别测量了正确调用率。到深度6时,模型自主运行下的能力损失约70%(L6=0.686, 0.684)。核心发现在于评估机制本身:在精确匹配评分规则下,传播模型的严重性与恢复能力参数被评分规则固定——严重性被强制为0(869个污染步骤中无一正确),恢复能力则结构上不可观测(580个污染步骤中未返回正轨,预期随机值为0.0058)。原因在于分歧后,黄金轨迹由模型从未见过的工具常量生成,信息无法推导。强行拟合却给出0.92和0.73的高置信度估计,实则已被评分规则决定。本文揭示该机制并提出补救方案:状态条件评分法,可对缓存结果回溯应用,无需额外成本,使严重性估计从0升至+0.149,恢复性升至+0.316。

原文摘要 · Abstract (English)

Tool-using agents fail two ways: choosing the wrong tool, or forming wrong arguments, and an early failure of either kind can silently corrupt everything downstream. We measure a correct-invocation rate that separates the two, under both a clean teacher-forced context and the model's own free-running context, on five open-weight models over contamination-free multi-step tasks (depths 1-8). By depth 6, roughly 70% of a model's own clean-context capability is lost to its own earlier mistakes (L6 = 0.686, 0.684). Our central finding concerns the measurement itself. Under exact-match scoring against a fixed gold trajectory, a propagation model's severity and recovery parameters are not merely hard to estimate - they are fixed by the scoring rule. Severity is forced to its boundary (0 of 869 poisoned steps correct); recovery is structurally unobservable (0 of 580 poisoned steps returned on-track, against an expected 0.0058 by chance). Both follow from one mechanism: post-divergence, the gold value is generated by tool constants the model never sees, so it is information the model cannot derive. A fit run anyway returns 0.92 and 0.73 for a quantity that is exactly 1.000 - confident numbers for a parameter the scoring rule already determined. We give the mechanism and a remedy, conditional-on-state scoring, applied retrospectively to cached completions at zero additional cost, which un-pins severity to interior estimates excluding zero (+0.149, +0.316).

智能体评估工具调用可靠性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。