发现代码智能体正确编码后仍失败的核心原因：代码一致性崩溃
Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code
- 分解智能体行为为搜索、读取、编辑三阶段，定位失败根源
- 60-69%的失败发生在正确定位后，因代码被错误覆盖或混乱修改
- 提出新评估方法，可识别并修复关键失败模式，适合模型开发者参考
代码智能体在SWE-bench Verified上解决率达65-70%，但Pass@1无法揭示剩余失败原因。我们发现，仅依赖最终结果会系统性误判高性能模型的失败。引入TRAJEVAL，一种无需训练的轨迹分解方法，将16,758条轨迹划分为参考补丁对齐的搜索、读取、编辑阶段。结果显示，60-69%的失败发生在已定位正确函数的情况下，且这一模式在SWE-Agent和OpenHands中普遍出现，且在bash-only LiveSWEAgent上持续存在。在这些情况下，我们识别出‘一致性崩溃’（Coherence Collapse）——即智能体达到正确代码后又将其覆盖或破坏，是主要失败模式，并在SWE-bench Verified与多语言PolyBench Verified中重复验证。5个案例中，智能体生成了与标准补丁完全一致的中间版本，随后被破坏；通过编辑提交检查点可恢复全部5例。一种无参考的共识驱动变体在GPT-5上带来+3.0个百分点的定向提升（p=0.08）。
原文摘要 · Abstract (English)
Code agents resolve 65-70% of SWE-bench Verified issues, but Pass@1 cannot tell us why the rest fail, and, as we show, capable-model failures are systematically misdiagnosed without trajectory data. We introduce TRAJEVAL, a training-free decomposition of agent trajectories into reference-patch-aligned search, read, and edit stages, and apply it across 16,758 trajectories spanning three architectures and seven models. The dominant failure of capable models is not localization: 60-69% of failures on SWE-Agent and OpenHands reach and edit the correct functions yet still produce incorrect patches, and the pattern persists for most models on the bash-only LiveSWEAgent. Within this Edit-Quality residual, we identify Coherence Collapse, where the agent reaches correct code and then overwrites or thrashes it, as the largest theme, replicating across SWE-bench Verified and the multilingual PolyBench Verified. In 5 cases, the agent produces a patch bit-identical to the gold reference mid-trajectory and destroys it later; an edit-commit checkpoint recovers all 5 against the SWE-bench Docker harness. A reference-free consensus-driven variant yields a directional +3.0 pp Pass@1 measurement on GPT-5 (p=0.08).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。