用推理型大模型分析代码差异,自动判断修复补丁是否真正解决问题。
RePaCA: Leveraging Reasoning Large Language Models for Static Automated Patch Correctness Assessment
- 让大模型通过思维链分析修复前后代码,判断是否解决根本问题。
- 在Defects4J数据集上达到83.1%准确率和84.8%F1分数,优于现有方法。
- 不仅更准,还能解释判断依据,适合需要可信修复验证的开发者。
自动化程序修复(APR)旨在无需人工干预的情况下自动修正软件缺陷。然而,现有工具生成的补丁常仅通过测试用例,未真正修复问题,称为过拟合补丁。为此,自动化补丁正确性评估(APCA)尝试识别此类过拟合补丁。静态APCA方法仅需原始代码与修复代码,无需额外信息。当前静态方法普遍存在可靠性、灵活性和透明性不足的问题。为此,本文提出RePaCA,一种利用专长于推理任务的大语言模型(LLM)的新型静态APCA技术。模型接收有缺陷代码与修复代码,被引导生成思维链,分析代码差异,推理补丁如何解决根本原因,并输出二分类结果:正确或过拟合。为增强该任务上的推理能力,采用基于组相对策略优化的强化学习对模型进行微调。在标准Defects4J衍生数据集上,本方法实现83.1%准确率与84.8% F1分数,达到当前最优。此外,模型在不同数据集间表现出更强泛化能力,优于领先技术。其推理过程还提升了评估结果的可解释性。这些发现表明,经微调的推理型大模型在提升静态APCA的准确性、泛化性和可解释性方面具有巨大潜力。
原文摘要 · Abstract (English)
Automated Program Repair (APR) seeks to automatically correct software bugs without requiring human intervention. However, existing tools tend to generate patches that satisfy test cases without fixing the underlying bug, those are known as overfitting patches. To address this issue, Automated Patch Correctness Assessment (APCA) attempts to identify overfitting patches generated by APR tools. It can be solved as a static approach, meaning that no additional information is needed beyond the original and fixed code snippets. Current static techniques often struggle with reliability, flexibility and transparency. To address these issues, we introduce RePaCA, a novel static APCA technique that leverages Large Language Models (LLMs) specialized in thinking tasks. Our model is prompted with both buggy and fixed code snippets and guided to generate a Chain of Thought that analyses code differences, reasons about how the patch addresses the root cause, and ultimately provides a binary classification: correct or overfitting. To enhance these reasoning capabilities for the APCA task specifically, the LLM is finetuned using Reinforcement Learning with the Group Relative Policy Optimization algorithm. When evaluated on a standard Defects4J-derived test, our approach achieves state-of-the-art performance, with 83.1% accuracy and an 84.8% F1-score. Furthermore, our model demonstrates superior generalization capabilities when trained on different datasets, outperforming the leading technique. This reasoning capability also provides enhanced explainability for the patch assessment. These findings underscore the considerable promise of finetuned, reasoning LLMs to advance static APCA by enhancing accuracy, generalization, and explainability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。