arXiv:2502.02747cs.ROcs.AI2025-02ICML被引 25

PatchPilot用规则流程实现高效稳定修复,成本低于1美元/实例。

PatchPilot: A Cost-Efficient Software Engineering Agent with Early Attempts on Formal Verification

  • 采用五步规则流程:复现、定位、生成、验证、优化,提升稳定性。
  • 在SWE-bench上表现优于开源方法,单次修复成本不足1美元。
  • 适合追求低成本高可靠性的软件自动化修复场景。

近期研究构建了多种结合大语言模型(LLMs)与非机器学习工具的修补代理,在当前最先进的软件修补基准SWE-bench上取得良好效果。现有修补代理可分为基于代理的规划方法(依赖LLM进行规划)和基于规则的规划方法(遵循预定义流程)。前者性能高但成本高且稳定性差,后者更稳定高效但关键流程受限导致修补效果下降。本文提出PatchPilot,一种兼顾修补效能、稳定性和成本效率的智能修补代理。PatchPilot设计了一种包含五个环节的新颖规则流程:复现、定位、生成、验证和优化(其中优化为独特设计),并在每个环节引入定制化改进以提升效果与效率。在SWE-bench基准上的大量实验表明,PatchPilot性能优于现有开源方法,同时保持极低成本(每实例少于1美元)并具备更高稳定性。我们还进行了详细的消融实验,验证各组件的关键设计有效性。代码已开源:https://github.com/ucsb-mlsec/PatchPilot。

原文摘要 · Abstract (English)

Recent research builds various patching agents that combine large language models (LLMs) with non-ML tools and achieve promising results on the state-of-the-art (SOTA) software patching benchmark, SWE-bench. Based on how to determine the patching workflows, existing patching agents can be categorized as agent-based planning methods, which rely on LLMs for planning, and rule-based planning methods, which follow a pre-defined workflow. At a high level, agent-based planning methods achieve high patching performance but with a high cost and limited stability. Rule-based planning methods, on the other hand, are more stable and efficient but have key workflow limitations that compromise their patching performance. In this paper, we propose PatchPilot, an agentic patcher that strikes a balance between patching efficacy, stability, and cost-efficiency. PatchPilot proposes a novel rule-based planning workflow with five components: reproduction, localization, generation, validation, and refinement (where refinement is unique to PatchPilot). We introduce novel and customized designs to each component to optimize their effectiveness and efficiency. Through extensive experiments on the SWE-bench benchmarks, PatchPilot shows a superior performance than existing open-source methods while maintaining low cost (less than 1$ per instance) and ensuring higher stability. We also conduct a detailed ablation study to validate the key designs in each component. Our code is available at https://github.com/ucsb-mlsec/PatchPilot.

代码修复智能代理成本优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。