首个大规模多语言代码调试基准,覆盖18种语言3600个测试样本。
MdEval: Massively Multilingual Code Debugging
- 构建包含18种语言的3600个代码缺陷样本的多语言调试数据集。
- 发现开源模型与闭源模型在多语言调试上存在显著性能差距。
- 适合研究多语言代码生成、自动修复及跨语言编程能力的开发者。
代码大语言模型(LLMs)在基于错误代码片段直接生成正确代码方面取得显著进展。现有编程评测基准多集中于Python,且语言多样性有限(如DebugBench和DebugEval)。为推动多语言代码调试发展,我们提出首个大规模多语言调试基准MDEVAL,涵盖18种编程语言的3600个测试样本,覆盖自动化程序修复(APR)、代码审查(CR)和缺陷识别(BI)任务。同时,我们构建了调试指令语料库MDEVAL-INSTRUCT,通过在正确多语言查询与解法中注入错误(xDebugGen)生成训练数据。进一步,训练了多语言调试器xDebugCoder作为强基线,专门处理多种语言中的典型错误(如Rust的'Missing Mut'和C语言的'Misused Macro Definition')。在MDEVAL上的大量实验显示,开源模型与闭源模型(如GPT和Claude系列)之间存在显著性能差距,表明多语言代码调试仍有巨大提升空间。
原文摘要 · Abstract (English)
Code large language models (LLMs) have made significant progress in code debugging by directly generating the correct code based on the buggy code snippet. Programming benchmarks, typically consisting of buggy code snippet and their associated test cases, are used to assess the debugging capabilities of LLMs. However, many existing benchmarks primarily focus on Python and are often limited in terms of language diversity (e.g., DebugBench and DebugEval). To advance the field of multilingual debugging with LLMs, we propose the first massively multilingual debugging benchmark, which includes 3.6K test samples of 18 programming languages and covers the automated program repair (APR) task, the code review (CR) task, and the bug identification (BI) task. Further, we introduce the debugging instruction corpora MDEVAL-INSTRUCT by injecting bugs into the correct multilingual queries and solutions (xDebugGen). Further, a multilingual debugger xDebugCoder trained on MDEVAL-INSTRUCT as a strong baseline specifically to handle the bugs of a wide range of programming languages (e.g. "Missing Mut" in language Rust and "Misused Macro Definition" in language C). Our extensive experiments on MDEVAL reveal a notable performance gap between open-source models and closed-source LLMs (e.g., GPT and Claude series), highlighting huge room for improvement in multilingual code debugging scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。