arXiv:2512.07850cs.LGcs.AI2025-12被引 14

小动作大错误:识别并修复大模型代理中关键失误步骤

SABER: Small Actions, Big Errors -- Safeguarding Mutating Steps in LLM Agents

  • 区分环境改变与非改变动作,定位导致失败的致命偏差
  • 每多一个环境改变步骤的偏差,成功概率下降高达96%
  • 提出无梯度、测试时可用的防护机制,提升复杂任务成功率

尽管大语言模型代理进展迅速,其在长周期、工具使用任务上的表现仍脆弱。我们提出核心问题:所有动作对失败的影响是否相同?在τ-Bench(Airline/Retail)和SWE-Bench Verified上分析执行轨迹,将路径分解为环境改变(mutating)与非环境改变步骤,并定义‘决定性偏差’——最早使成功转为失败的动作级别偏离。逻辑回归显示,每个额外的环境改变步骤偏差,使成功几率在Airline任务上降低最多92%,Retail任务上降低最多96%。而非环境改变步骤的偏差影响微乎其微。错误随上下文长度增长,因代理偏离角色并基于过时约束行动。基于此,我们提出SABER,一种模型无关、无梯度、测试时可用的防护机制:(i) 添加仅在环境改变时激活的验证,(ii) 在环境改变前注入‘目标反思’,(iii) 进行分块上下文清理。SABER带来稳定提升,如Qwen3-Thinking在Airline上相对提升28%,Retail上11%,SWE-Bench Verified上7%;Claude分别提升9%和7%。我们还发现τ-Bench存在天花板效应,因标注错误和任务描述不明确人为限制模型性能。为此,我们发布τ-Bench Verified,通过针对性修订恢复基准评估空间。结果强调:需进行动作级分析、实施精准防护、保障可靠评估,方能实现稳健的多轮代理。

原文摘要 · Abstract (English)

Despite rapid progress in LLM agents, performance on long-horizon, tool-using tasks remains fragile. To better understand this fragility, we ask a simple question: \emph{do all actions contribute equally to failure?} Analyzing execution traces on $τ$-Bench (Airline/Retail) and SWE-Bench Verified, we decompose trajectories into \emph{mutating} (environment-changing) vs.\ non-mutating steps and formalize \emph{decisive deviations}, earliest action, level divergences that flip success to failure. A logistic regression reveals that each additional deviation in a mutating action reduces the odds of success by upto $92\%$ on Airline and upto $96\%$ on Retail for SoTA models. In contrast, deviations in non-mutating actions have little to no effect. Errors also grow with context length as agents drift from role and act on stale constraints. Motivated by these observations, we introduce \cm{}, a model-agnostic, gradient-free, test-time safeguard that (i) adds mutation-gated verification, (ii) injects \emph{Targeted Reflection} before mutating steps, and (iii) performs block-based context cleaning. \cm{} delivers consistent gains, e.g., Qwen3-Thinking: +28\% \emph{relative} on Airline, +11\% on Retail, and +7\% on SWE-Bench Verified; Claude: +9\%/+7\%. We further identify ceiling effects in $τ$-Bench, where annotation errors and underspecified tasks artificially cap model performance. To address this, we release $τ$-Bench Verified, which restores benchmark headroom through targeted revisions. Our results argue for action-level analysis, targeted safeguards, and reliable evaluations as prerequisites for robust multi-turn agents.

大模型代理错误分析任务可靠性测试时防护

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。