用大模型生成1.2万条真实代码漏洞，构建更难的漏洞修复评估基准。
Diff-Based Code Corruption using LLMs for Large-Scale Bugfix Benchmarking
- 用大模型生成代码差异（diff）注入漏洞，避免简单或无效修改。
- 13个开源模型在新基准上表现普遍下降，验证其挑战性。
- 适合研究模型真实修复能力，或开发更鲁棒的代码生成系统者使用。
现有漏洞修复评测基准大多规模小、漏洞类型单一，难以反映真实场景。为此，我们提出MegaBugFix，一个包含12,629个由正确Python程序通过大模型合成的有缺陷程序的大型基准。漏洞通过代表代码变更的diff方式生成，有效避免了传统方法中出现的过于简单或未真正修改程序的问题。我们在MegaBugFix及基线基准上评估了13个开放权重模型，结果均显示性能显著下降，说明该基准能揭示现有评测中隐藏的模型缺陷。该基准及用于漏洞注入的微调模型已公开于hf.co/collections/szalontaib/megabugfix。
原文摘要 · Abstract (English)
There are various benchmarks to evaluate bugfixing capabilities of Large Language Models. However, most widespread benchmarks do not fully reflect real-world bugfixing practices. They are small, weakening statistical reliability, and the buggy programs are often similar to one another, potentially distorting evaluation results. The range of bug types can also be narrow, failing to capture a representative range of bugs. To address these issues, we introduce MegaBugFix, a large-scale bugfixing benchmark containing 12,629 buggy Python programs synthesized from correct ones by a Large Language Model. Bug injections were generated as diffs representing code changes. Through this approach, we were able to avoid common pitfalls of LLM-based mutation techniques like injecting overly simplistic bugs or failing to modify the input program. We evaluated 13 open-weight models on MegaBugFix and baseline benchmarks, finding consistently lower performance on MegaBugFix. This reveals that our benchmark presents more challenging bugs and exposes model failures that may remain hidden when evaluating on existing benchmarks. The benchmark and fine-tuned model used for bug injection are available at hf.co/collections/szalontaib/megabugfix.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。