测试大模型在代码修复中的泛化能力，发现效果大幅下降。
Evaluating the Generalizability of LLMs in Automated Program Repair
- 用变换后的Defects4J数据集测试11个大模型的修复能力
- 新数据集上正确和合理修复方案减少超四成，平均降49.48%
- 增加提示信息可提升效果，但仍无法恢复原始性能
基于大语言模型的自动化程序修复方法虽表现优异，但主要在Defects4J等少数数据集上评估，其在新数据上的有效性存疑。本研究在由Defects4J转换而来且保持语义一致的新数据集DEFECTS4J-TRANS上，评估了11个顶尖大模型的表现。实验显示，所有模型在新数据集上的泛化能力有限，平均正确修复补丁数下降49.48%，合理补丁数下降42.90%。进一步分析表明，向修复提示中加入更多相关上下文信息可显著提升模型能力（正确补丁最多提升136.67%，合理补丁最多提升121.82%），但性能仍远低于原水平。这说明仅靠提示工程无法根本提升大模型的修复能力。研究据此提出未来改进方向。
原文摘要 · Abstract (English)
LLM-based automated program repair methods have attracted significant attention for their state-of-the-art performance. However, they were primarily evaluated on a few well known datasets like Defects4J, raising questions about their effectiveness on new datasets. In this study, we evaluate 11 top-performing LLMs on DEFECTS4J-TRANS, a new dataset derived from transforming Defects4J while maintaining the original semantics. Results from experiments on both Defects4J and DEFECTS4J-TRANS show that all studied LLMs have limited generalizability in APR tasks, with the average number of correct and plausible patches decreasing by 49.48% and 42.90%, respectively, on DEFECTS4J-TRANS. Further investigation into incorporating additional repair-relevant information in repair prompts reveals that, although this information significantly enhances the LLMs' capabilities (increasing the number of correct and plausible patches by up to 136.67% and 121.82%, respectively), performance still falls short of their original results. This indicates that prompt engineering alone is insufficient to substantially enhance LLMs' repair capabilities. Based on our study, we also offer several recommendations for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。