用指令微调模型优化代码修复,少样本也能大幅提升修复效率。
The Art of Repair: Optimizing Iterative Program Repair with Instruction-Tuned Models
- 用少量数据微调大模型,生成更多有效修复方案。
- 仅限10个补丁时,迭代修复比一次性生成效果好3倍以上。
- 复杂漏洞修复中,迭代策略优势更明显,适合高难度任务。
自动程序修复(APR)旨在减少手动排查和修复代码错误的工作量。在大型语言模型(LLM)兴起前,常见做法是生成数千个补丁以提升修复效果。近年来,自迭代能力使LLM能通过反馈多轮优化补丁。但现有研究多关注大量迭代,忽视输出数量的影响。本文构建了一个平衡机制:每缺陷最多生成10个补丁,结合多轮迭代。我们使用DeepSeekCoder-Instruct、Codellama-Instruct、Llama3.1-Instruct三个SOTA指令微调模型,在1K、30K、65K规模的APR数据集上分别采用全量微调与LoRA进行训练,并在HumanEval-Java和Defects4J两个基准上评估。结果表明,仅用不足1%的微调数据,即可实现可修复补丁数最高78%的提升,挑战了此前认为全量微调增益有限的观点。然而,超过一定阈值后性能开始下降,可能因过拟合。此外,基础模型通过迭代修复显著优于一次性生成;在复杂基准上,迭代优势更加突出。即使微调模型也从迭代中获益,尤其在复杂漏洞上。研究强调应平衡多输出生成与迭代精炼策略。
原文摘要 · Abstract (English)
Automatic program repair (APR) aims to reduce the manual efforts required to identify and fix errors in source code. Before the rise of LLM-based agents, a common strategy was to increase the number of generated patches, sometimes to the thousands, to achieve better repair results on benchmarks. More recently, self-iterative capabilities enabled LLMs to refine patches over multiple rounds guided by feedback. However, literature often focuses on many iterations and disregards different numbers of outputs. We investigate an APR pipeline that balances these two approaches, the generation of multiple outputs and multiple rounds of iteration, while imposing a limit of 10 total patches per bug. We apply three SOTA instruction-tuned LLMs - DeepSeekCoder-Instruct, Codellama-Instruct, Llama3.1-Instruct - to the APR task. We further fine-tune each model on an APR dataset with three sizes (1K, 30K, 65K) and two techniques (Full Fine-Tuning and LoRA), allowing us to assess their repair capabilities on two APR benchmarks: HumanEval-Java and Defects4J. Our results show that by using only a fraction (<1%) of the fine-tuning dataset, we can achieve improvements of up to 78% in the number of plausible patches generated, challenging prior studies that reported limited gains using Full Fine-Tuning. However, we find that exceeding certain thresholds leads to diminishing outcomes, likely due to overfitting. Moreover, we show that base models greatly benefit from creating patches in an iterative fashion rather than generating them all at once. In addition, the benefit of iterative strategies becomes more pronounced in complex benchmarks. Even fine-tuned models, while benefiting less from iterations, still gain advantages, particularly on complex benchmarks. The research underscores the need for balanced APR strategies that combine multi-output generation and iterative refinement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。