现有代码编辑评测基准与真实开发场景严重脱节，亟需更贴近实际的评估体系。
Edit, But Verify: An Empirical Audit of Instructed Code-Editing Benchmarks
- 对比真实开发数据，发现两大评测集仅覆盖少数语言和场景。
- 多数题目测试用例不足，超半数无法检测非修改区域的错误。
- 提出六项改进标准，公开审计数据供社区共建可靠评测体系。
指令式代码编辑——即大模型根据自然语言指令修改已有代码——占实际编程助手交互的约19%。然而，现有代码相关基准中仅有两个（CanItEdit 和 EDIT-Bench）专门针对指令式代码编辑，且采用人工编写指令与基于测试的评估。我们通过对比其编程语言、编辑意图与应用领域，发现它们在分布上与真实世界（Copilot Arena、AIDev、GitHub Octoverse）严重偏离：超过90%的评测集中在Python，TypeScript（GitHub最常用语言）完全缺失；后端与前端开发（占46%真实编辑活动）基本缺席；文档、测试与维护类编辑（占31.4%的人类PR）无任何体现。两基准测试用例数量有限（CanItEdit中位数13，EDIT-Bench中位数4），且59%的低覆盖率测试套件无法检测修改区外的错误。其中15个问题被40个大模型全部失败，11个源于评测数据缺陷而非模型能力。此外，29%的EDIT-Bench问题与至少一个其他问题共享代码库，6%的CanItEdit问题存在类似情况。综上，当前基准所测能力远窄于部署需求。因此我们提出六项基于实证的优化原则，并发布全部审计数据，助力构建真正反映真实编辑能力的评测体系。
原文摘要 · Abstract (English)
Instructed code editing, where an LLM modifies existing code based on a natural language instruction, accounts for roughly 19% of real-world coding assistant interactions. Yet very few benchmarks directly evaluate this capability. From a survey of over 150 code-related benchmarks, we find that only two, CanItEdit and EDIT-Bench, target instructed code editing with human-authored instructions and test-based evaluation. We audit both by comparing their programming languages, edit intents, and application domains against distributions observed in the wild (Copilot Arena, AIDev, GitHub Octoverse), and by measuring test counts, statement coverage, and test scope across all 213 problems. Both benchmarks concentrate over 90\% of evaluation on Python while TypeScript, GitHub's most-used language, is absent. Backend and frontend development, which together constitute 46% of real-world editing activity, are largely missing, and documentation, testing, and maintenance edits (31.4% of human PRs) have zero representation. Both benchmarks have modest test counts (CanItEdit median 13, EDIT-Bench median 4), though CanItEdit compensates with near-complete whole-file coverage and fail-before/pass-after validation. 59\% of EDIT-Bench's low-coverage suites would not detect modifications outside the edit region. EDIT-Bench has 15 problems that are not solved by any of 40 LLMs and 11 of these problems trace failures to poor benchmark artifacts rather than model limitations. Further, 29% of EDIT-Bench problems and 6% of CanItEdit problems share a codebase with at least one other problem within the benchmark. In summary, these benchmarks measure a narrower construct than deployment decisions require. We therefore propose six empirically grounded desiderata and release all audit artifacts so the community can build instructed code-editing benchmarks whose scores reliably reflect real-world editing capability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。