arXiv:2509.13941cs.SEcs.AI2025-09被引 19

分析代码自动修复失败原因,提出协作框架提升成功率

An Empirical Study on Failures in Automated Issue Solving

  • 通过人工分析150个失败案例,建立三阶段九类失败分类体系
  • 发现代理型模型多因推理错误和认知死锁导致失败,占比超60%
  • 设计专家-执行者协同框架,使难题解决率提升22.2%

自动化问题求解旨在自主识别并修复整个代码库中的缺陷代码片段。SWE-Bench已成为评估该领域进展最广泛使用的基准。尽管基于大语言模型的代理工具展现出巨大潜力,但在大量任务上仍会失败。当前评估主要报告整体修复率,掩盖了成功与失败的根本原因,难以诊断模型弱点或指导针对性改进。为此,我们首先分析三种前沿工具在不同任务特征下的表现与效率,涵盖基于流水线与代理架构的模型。进一步地,为实现从宏观指标到根本原因的分析,我们对150个失败实例进行了系统性人工分析,构建了一个包含三个主阶段、九个主类别、二十五个细粒度子类别的失败模式分类体系。系统分析表明,两种架构范式存在明显差异,代理型模型的失败主要源于错误推理与认知死锁。基于此洞察,我们提出一种协作式专家-执行者框架,引入监督专家代理对主执行代理进行战略监控与纠偏。该架构旨在纠正推理偏差并打破认知死锁。实验显示,该框架可解决此前单代理无法处理的22.2%的问题,为构建更鲁棒的智能体提供了诊断评估与协作设计的新路径。

原文摘要 · Abstract (English)

Automated issue solving seeks to autonomously identify and repair defective code snippets across an entire codebase. SWE-Bench has emerged as the most widely adopted benchmark for evaluating progress in this area. While LLM-based agentic tools show great promise, they still fail on a substantial portion of tasks. Moreover, current evaluations primarily report aggregate issue-solving rates, which obscure the underlying causes of success and failure, making it challenging to diagnose model weaknesses or guide targeted improvements. To bridge this gap, we first analyze the performance and efficiency of three SOTA tools, spanning both pipeline-based and agentic architectures, in automated issue solving tasks of SWE-Bench-Verified under varying task characteristics. Furthermore, to move from high-level performance metrics to underlying cause analysis, we conducted a systematic manual analysis of 150 failed instances. From this analysis, we developed a comprehensive taxonomy of failure modes comprising 3 primary phases, 9 main categories, and 25 fine-grained subcategories. Then we systematically analyze the distribution of the identified failure modes, the results reveal distinct failure fingerprints between the two architectural paradigms, with the majority of agentic failures stemming from flawed reasoning and cognitive deadlocks. Motivated by these insights, we propose a collaborative Expert-Executor framework. It introduces a supervisory Expert agent tasked with providing strategic oversight and course-correction for a primary Executor agent. This architecture is designed to correct flawed reasoning and break the cognitive deadlocks that frequently lead to failure. Experiments show that our framework solves 22.2% of previously intractable issues for a leading single agent. These findings pave the way for building more robust agents through diagnostic evaluation and collaborative design.

自动化修复大模型评测失败分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。