测试大模型修复真实项目可维护性问题的效果，发现修复率约三成，但常引入新错误。
Evaluating the Effectiveness of LLMs in Fixing Maintainability Issues in Real-World Projects
- 用零样本和少样本提示评估Copilot Chat与Llama修复代码可维护性问题。
- 少样本提示下Llama修复成功率44.9%，零样本下仅为30%~32.3%。
- 虽部分代码可读性提升，但多数修复方案引入编译错误或新问题。
大型语言模型（LLMs）在解决编程问题上备受关注，但其在修复代码可维护性方面的效果尚不明确。本研究评估了LLM在10个GitHub仓库中修复127个可维护性问题的能力。采用零样本提示对Copilot Chat和Llama 3.1进行测试，对Llama使用少样本提示。评估内容包括编译错误、测试失败及新产生的可维护性问题。结果显示，使用少样本提示的Llama成功修复44.9%的方法，而Copilot Chat和Llama零样本提示分别修复32.29%和30%。然而，大多数生成解决方案引入了错误或新的可维护性问题。此外，通过45名参与者的人类研究评估了51个LLM生成方案的可读性，结果显示68.63%的参与者认为可读性有所改善。总体而言，尽管LLM在修复可维护性问题方面展现出潜力，但其引入错误的倾向凸显了当前技术的局限性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have gained attention for addressing coding problems, but their effectiveness in fixing code maintainability remains unclear. This study evaluates LLMs capability to resolve 127 maintainability issues from 10 GitHub repositories. We use zero-shot prompting for Copilot Chat and Llama 3.1, and few-shot prompting with Llama only. The LLM-generated solutions are assessed for compilation errors, test failures, and new maintainability problems. Llama with few-shot prompting successfully fixed 44.9% of the methods, while Copilot Chat and Llama zero-shot fixed 32.29% and 30%, respectively. However, most solutions introduced errors or new maintainability issues. We also conducted a human study with 45 participants to evaluate the readability of 51 LLM-generated solutions. The human study showed that 68.63% of participants observed improved readability. Overall, while LLMs show potential for fixing maintainability issues, their introduction of errors highlights their current limitations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。