用智能体自动修复大规模代码测试失败，结合静态分析与测试反馈。
Agentic Program Repair from Test Failures at Scale: A Neuro-symbolic approach with static analysis and test execution feedback
- 基于ReAct框架和LLM构建智能体，通过读文件、生成补丁等15步动作修复代码。
- 在离线测试中解决率42.3%，平均需11.8轮反馈迭代，70B模型性能媲美405B大模型。
- 生产环境中31.5%修复被采纳，工程师反馈积极，部分结果可作优化起点。
随着大语言模型的发展，大型组织在大规模代码库上实现智能体式程序修复成为可能。本文开发了一个工程智能体，基于内部多产品线的测试失败，在大规模范围内自动修复源码。以Llama为基础，采用ReAct框架构建智能体，从规则化测试失败机器人识别的错误出发，通过15个动作（如读取文件、生成补丁）执行修复流程，并利用静态分析工具和测试执行轨迹提供反馈以优化方案。引入LLM作为评审员，确保补丁符合人工审查标准。离线评估显示，专用70B模型性能接近更庞大的405B模型；消融实验证明，静态分析与测试反馈显著提升神经模型表现。在基准测试中，平衡求解率与错误率的模型平均需11.8轮反馈，解决率达42.3%。生产阶段三个月内，80%生成修复被评审，其中31.5%被合并（占总生成数的25.5%）。工程师反馈通过开放式编码分析，呈现快速批准、感激与惊喜情绪，亦有对部分正确但不完整修复的混合评价，认为其适合作为优化起点。
原文摘要 · Abstract (English)
Aim: With the advent of LLMs, sophisticated agentic program repair has become viable at large organizations with large codebases. In this work, we develop an Engineering Agent that fixes the source code based on test failures at scale across diverse software offerings internally. Method: Using Llama as the base, we employ the ReAct harness to develop an agent. We start with a test failure that was triaged by a rule-based test failure bot. We then set up an agentic harness and allow the agent to reason and run a set of 15 actions from reading a file to generating a patch. We provide feedback to the agent through static analysis and test failures so it can refine its solution. We leverage an LLM-as-a-Judge to ensure that the patch conforms to the standards followed by a human review to land fixes. Benchmark Findings: We curated offline benchmarks for our patch generator, the Engineering Agent loop, and the LLM-as-a-Judge. In offline evaluations we found that a specialized 70B model is highly competitive with the much larger but vanilla Llama-405B. In an ablation study, we found that the ReAct harness (neural model) benefited from the symbolic information from static analysis tools and test execution traces. A model that strikes a balance between the solve rate and error rate vs the cost and latency has a benchmark solve rate of 42.3% using an average 11.8 feedback iterations. Production Findings: In a three month period, 80% of the generated fixes were reviewed, of which 31.5% were landed (25.5% of the total number of generated fixes). Feedback from Engineers: We used open coding to extract qualitative themes from engineers' feedback. We saw positive feedback in the form of quick approvals, gratitude, and surprise. We also found mixed feedback when the Engineering Agent's solution was partially correct and it served as a good starting point.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。