构建多任务多语言代码库级调试数据集，评估大模型真实场景修复能力
RepoDebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models
- 构建涵盖22类错误、8种语言的代码库级调试数据集
- 10个大模型在该数据集上表现普遍不佳，最佳模型仍不理想
- 填补了代码修复评估从函数级到真实项目级的空白
大型语言模型（LLMs）在代码调试方面展现出显著能力，尤其在自动程序修复上，有望大幅减少开发者耗时并提升效率。尽管调试数据集已取得显著进展，但现有数据集主要聚焦于函数级别的代码修复能力评估，忽略了更复杂且贴近实际的代码库级场景，导致对大模型在代码库级调试中面临挑战的理解不完整。虽然已有若干代码库级数据集被提出，但普遍存在任务、语言和错误类型多样性不足的问题。为此，本文提出RepoDebug，一个支持8种常用编程语言、3种调试任务和22种错误子类型的多任务、多语言代码库级代码调试数据集。我们还在10个大模型上进行了评估实验，结果显示，表现最好的模型Claude 3.5 Sonnect在代码库级调试任务中仍表现不佳。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have exhibited significant proficiency in code debugging, especially in automatic program repair, which may substantially reduce the time consumption of developers and enhance their efficiency. Significant advancements in debugging datasets have been made to promote the development of code debugging. However, these datasets primarily focus on assessing the LLM's function-level code repair capabilities, neglecting the more complex and realistic repository-level scenarios, which leads to an incomplete understanding of the LLM's challenges in repository-level debugging. While several repository-level datasets have been proposed, they often suffer from limitations such as limited diversity of tasks, languages, and error types. To mitigate this challenge, this paper introduces RepoDebug, a multi-task and multi-language repository-level code debugging dataset with 22 subtypes of errors that supports 8 commonly used programming languages and 3 debugging tasks. Furthermore, we conduct evaluation experiments on 10 LLMs, where Claude 3.5 Sonnect, the best-performing model, still cannot perform well in repository-level debugging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。