用上下文信息增强LLM修复回归缺陷，成功率提升1.6倍
From Empirical Evaluation to Context-Aware Enhancement: Repairing Regression Errors with LLMs
- 引入变更上下文信息提升LLM修复回归错误能力
- 相比无上下文的LLM方法，修复成功率提高1.6倍
- 在Java和Python上均有效，适合自动化修复研究者
自那以来，各种自动程序修复（APR）方法，尤其是利用大语言模型（LLM）能力的方法，迅速发展以修复通用软件缺陷。然而，这些先进方法在修复回归缺陷方面的有效性仍基本未被探索。这一空白促使我们开展一项实证研究，评估现代APR技术修复真实世界回归缺陷的效果。本文构建了RegressionBug4APR基准，包含从广泛使用的GitHub仓库中收集的200个高质量的Java和Python回归缺陷，并设计了支持APR研究的框架。我们对基准进行了深入分析，验证其多样性和质量。在此基础上，我们实证评估了传统APR工具和先进的基于LLM的APR方法在修复回归缺陷上的表现。实验结果表明，经典工具未能修复任何缺陷，而基于LLM的方法展现出良好潜力。受此启发，我们进一步研究将导致缺陷的代码变更信息融入基于LLM的APR方法的效果。通过消融实验，我们分解了变更信息中各上下文元素的贡献。结果表明，这种上下文感知增强显著提升了基于LLM的修复性能，相比不使用此类上下文的方法，成功修复数量提升1.6倍。此外，该结论在Java和Python基准上均一致，为发现的普适性提供了初步证据。
原文摘要 · Abstract (English)
[...] Since then, various APR approaches, especially those leveraging the power of large language models (LLMs), have been rapidly developed to fix general software bugs. Unfortunately, the effectiveness of these advanced techniques in the context of regression bugs remains largely unexplored. This gap motivates the need for an empirical study evaluating the effectiveness of modern APR techniques in fixing real-world regression bugs. In this work, we conduct an empirical study of APR techniques on regression bugs. To facilitate our study, we introduce RegressionBug4APR, a high-quality benchmark of Java and Python regression bugs integrated into a framework designed to facilitate APR research. The current benchmark includes 200 regression bugs collected from widely used real-world GitHub repositories. We begin by conducting an in-depth analysis of the benchmark, demonstrating its diversity and quality. Building on this foundation, we empirically evaluate the capabilities of APR to regression bugs by assessing both traditional APR tools and advanced LLM-based APR approaches. Our experimental results show that classical APR tools fail to repair any bugs, while LLM-based APR approaches exhibit promising potential. Motivated by these results, we investigate impact of incorporating bug-inducing change information into LLM-based APR approaches for fixing regression bugs. We further conduct an ablation study to disaggregate the contribution of each contextual element within the bug-inducing change information. Our results highlight that this context-aware enhancement significantly improves the performance of LLM-based APR, yielding 1.6x more successful repairs compared to using LLM-based APR without such context. Moreover, our findings are consistent across both Java and Python benchmarks, providing preliminary evidence for the generalizability of our findings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。