arXiv:2607.09510cs.SEcs.AI2026-07被引 4

首次系统分析代码代理失败过程,揭示其早期出现且难以恢复的特性。

Failure as a Process: An Anatomy of CLI Coding Agent Trajectories

论文配图:Failure as a Process: An Anatomy of CLI Coding Agent Trajectories
图 1 · 摘自论文原文
  • 将失败视为动态过程,分析其发生、演变与恢复轨迹
  • 发现90%以上失败源于认知错误,多在前几步产生
  • 适合关注代码生成可靠性与安全性的开发者与研究者

大型语言模型(LLM)代码代理在终端环境中日益用于自主执行软件工程任务,其可靠性成为关注焦点。现有研究多将失败视为终点结果,而非时间过程,难以揭示失败如何产生、演化并不可恢复。本文开展首个大规模实证研究,分析命令行接口(CLI)代码代理的失败轨迹,提出基于过程的分析框架,涵盖失败的起始、演化与恢复阶段。我们收集了七种前沿模型在三个代理架构(OpenHands、MiniSWE、Terminus2)上于Terminal-Bench生成的3,843条执行轨迹,经筛选获得1,794条完整有效轨迹(超过63,000个执行步骤),进行人工标注,得出14项发现,涵盖失败发生、根本原因、恢复能力及跨系统一致性。结果显示,代码代理失败主要由认知错误驱动,通常在前几步即开始,且常隐匿至无法恢复,提示提升可靠性需更早介入验证,而非仅依赖最终结果评估。

原文摘要 · Abstract (English)

Large language model (LLM) coding agents are increasingly deployed to autonomously perform software engineering tasks in terminal-based environments, making their reliability a growing concern. Existing empirical studies investigate why coding agents fail, yet they largely treat failure as a final outcome rather than a temporal process, providing limited insight into how failures emerge, evolve, and become unrecoverable. We present the first large-scale empirical study of CLI coding-agent failure trajectories, introducing a process-oriented framework that analyzes failure through its onset, evolution, and recovery across execution trajectories. We first collect 3,843 execution trajectories generated by seven frontier models across three coding-agent scaffolds (OpenHands, MiniSWE, and Terminus2) on Terminal-Bench, then carefully filter them to obtain 1,794 complete and valid trajectories for manual annotation (over 63,000 execution steps), from which we derive 14 findings spanning failure occurrence, root causes, recovery, and cross-system consistency. Our findings show that coding-agent failures are predominantly driven by epistemic errors, typically begin within the first few execution steps, and often remain hidden until recovery is no longer possible, suggesting that improving coding-agent reliability requires earlier validation and intervention rather than relying solely on final-outcome evaluation.

代码生成失败分析LLM可靠性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。