首次系统分析5个自动修复工具的决策过程,发现其在复杂逻辑错误上表现不佳。
Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study
- 追踪5个顶级自动修复工具在500个真实任务中的完整决策链
- 面对逻辑密集型缺陷时生成冗长且过拟合的补丁,仅通过测试但未解决问题根源
- 适合关注自动化修复局限性与下一代系统设计的研究者
自动化程序修复(APR)代理利用大语言模型(LLMs)进行自主诊断和修复软件缺陷,具备规划、推理与工具调用能力。尽管在SWE-bench等排行榜上表现优异,对其行为机制、失败原因及与人类开发者差异的理解仍十分有限。本文对5个最先进的APR代理进行了首次系统性分析,追踪其在500个真实修复任务中从问题描述到补丁验证的完整决策流程。研究发现,尽管代理在简单修复中表现良好,但在逻辑密集型缺陷上存在明显不足,常生成冗长且过拟合的补丁,虽通过现有测试套件但未能解决根本问题。测试生成与回归测试选择仍是主要瓶颈,代理难以重现问题或执行相关回归测试。此外,多数代理仅使用基础工具(如bash脚本),缺乏调试器或程序分析工具支持。这些发现揭示了当前APR系统的关键局限,并提出了下一代设计方向:(1)强调早期高质量测试生成与验证的左移策略;(2)构建更丰富集成的工具生态;(3)采用融合互补优势的多样化代理架构;(4)建立以语义修复质量与测试生成真实性为核心的新基准。
原文摘要 · Abstract (English)
Automated Program Repair (APR) agents leverage large language models (LLMs) to autonomously diagnose and patch software bugs using planning, reasoning, and tools. Although these agents show strong performance on leaderboards such as SWE-bench, little is understood about how they take actions, where they fail, and how their behavior compares to human developers. In this paper, we present the first systematic analysis of these limitations using 5 state-of-the-art APR agents. We trace the full decision-making pipelines of the 5 APR agents across 500 real-world repair tasks, from issue description to patch validation. Our study reveals that, while agents excel at simple fixes, they struggle with logic-intensive bugs, often generating verbose, overfitted patches that pass existing test suites without solving the root cause. Test generation and regression test selection remain major bottlenecks, as agents fail to reproduce issues or run relevant regression tests. Moreover, many agents operate with primitive tooling (e.g. bash scripts) and do not have access to debuggers or program analysis tools. These findings highlight key limitations of current APR systems and motivate several directions for next-generation APR design, including but not limited to: (1) a shift-left approach emphasizing early, high-quality test generation and validation to reduce spurious fixes and improve semantic correctness; (2) richer, more integrated tool ecosystems; (3) diversified agent architectures that combine complementary strengths; and (4) benchmarks that prioritize semantic repair quality and test-generation fidelity over surface-level success metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。