测试大模型调试能力，发现其常过度修改代码。
Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
- 自动生成带多错误的程序，精确评估修复动作
- 前沿模型修复准确率超76%但精准度低于45%
- 强调最小修改原则，适合研究代码生成与调试
与代码补全不同，调试需准确定位故障并实施针对性修改。我们观察到，前沿大模型在调试时经常生成正确但过度修改的解决方案。为评估大模型距离精确调试还有多远，我们提出精确调试基准（PDB）框架，可将任意编程数据集自动转换为具备精度感知评估的调试基准。PDB通过合成已验证的原子级错误，并将其组合成多错误程序来生成有缺陷代码。我们定义了两个新指标：编辑级精度和错误级召回率，分别衡量必要修改量与错误解决比例。我们发布了两个评估基准：PDB-Single-Hard（单行错误）和PDB-Multi（多行错误）。实验表明，如GPT-5.1-Codex和DeepSeek-V3.2-Thinking等前沿模型，单元测试通过率超过76%，但在明确要求最小化修改时，编辑精度仍低于45%。此外，迭代式与代理式调试策略也未显著提升精度或召回率，凸显需重新思考代码模型的后训练流程。
原文摘要 · Abstract (English)
Unlike code completion, debugging requires localizing faults and applying targeted edits. We observe that frontier LLMs often regenerate correct but over-edited solutions during debugging. To evaluate how far LLMs are from precise debugging, we introduce the Precise Debugging Benchmark (PDB) framework, which automatically converts any coding dataset into a debugging benchmark with precision-aware evaluation. PDB generates buggy programs by synthesizing verified atomic bugs and composing them into multi-bug programs. We define two novel metrics, edit-level precision and bug-level recall, which measures how many necessary edits are made and how many bugs are resolved. We release two evaluation benchmarks: PDB-Single-Hard on single-line bugs, and PDB-Multi on multi-line bugs. Experiments show that frontier models, such as GPT-5.1-Codex and DeepSeek-V3.2-Thinking, achieve unit-test pass rates above 76% but exhibit precision below 45%, even when explicitly instructed to perform minimal debugging. Finally, we show that iterative and agentic debugging strategies do not substantially improve precision or recall, highlighting the need to rethink post-training pipelines for coding models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。