GPT-o1在40个漏洞上全部修复，超越此前所有版本。
Can GPT-O1 Kill All Bugs? An Evaluation of GPT-Family LLMs on QuixBugs
- 对比GPT系列多个版本，测试其自动修复能力
- O1系列在QuixBugs上实现40/40成功修复率
- 适合关注大模型编程修复性能的开发者
大型语言模型在自动程序修复（APR）中表现出色，OpenAI的ChatGPT是该领域最常用模型之一。随着GPT系列模型持续迭代，其修复能力已达到领先水平。然而，针对不同版本GPT模型在APR中的表现比较研究仍较少。本文受近期GPT-o1模型公开启发，首次系统评估GPT家族多版本在APR中的效果。我们在QuixBugs基准上，从修复成功率、修复成本、响应长度和行为模式等多维度，对最新版本O1-preview、O1-mini、GPT-4o及历史版ChatGPT进行了实证分析。结果表明，O1系列修复能力显著优于以往版本，成功修复了全部40个漏洞。本研究为深入探索GPT系列模型在程序修复中的应用提供了基础。
原文摘要 · Abstract (English)
LLMs have long demonstrated remarkable effectiveness in automatic program repair (APR), with OpenAI's ChatGPT being one of the most widely used models in this domain. Through continuous iterations and upgrades of GPT-family models, their performance in fixing bugs has already reached state-of-the-art levels. However, there are few works comparing the effectiveness and variations of different versions of GPT-family models on APR. In this work, inspired by the recent public release of the GPT-o1 models, we conduct the first study to compare the effectiveness of different versions of the GPT-family models in APR. We evaluate the performance of the latest version of the GPT-family models (i.e., O1-preview and O1-mini), GPT-4o, and the historical version of ChatGPT on APR. We conduct an empirical study of the four GPT-family models against other LLMs and APR techniques on the QuixBugs benchmark from multiple evaluation perspectives, including repair success rate, repair cost, response length, and behavior patterns. The results demonstrate that O1's repair capability exceeds that of prior GPT-family models, successfully fixing all 40 bugs in the benchmark. Our work can serve as a foundation for further in-depth exploration of the applications of GPT-family models in APR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。