构建更真实、更严格的代码生成模型评测基准,发现模型在算法生成上表现差。
Code2Bench: Scaling Source and Rigor for Dynamic Benchmark Construction
- 从真实代码库动态获取题目,用自动化测试保证测试覆盖度
- 10个主流模型在新基准上表现差异大,算法题准确率显著低于调用接口题
- 揭示了旧基准中‘看似正确’的虚假表现,适合做代码生成评估的研究者
代码生成大模型的评估受限于静态易污染的题目来源和低严谨性的测试方式。本文提出双尺度基准构建方法,通过持续扩展真实代码仓库中的问题源,并以自动化高覆盖率的基于性质的测试(PBT)系统提升测试严谨性。我们实现了CODE2BENCH框架,利用作用域图分析进行依赖分类,并设置100%分支覆盖率质量门禁以保障测试集完整性。基于此框架,构建了包含2509个原生实例的CODE2BENCH-2509基准,涵盖Python与Java。对10个顶尖代码生成模型的全面评估显示:(1)模型存在根本性能差距,擅长调用接口任务(弱自包含),却在算法合成(自包含)任务中表现不佳;(2)目标语言生态深刻影响模型表现,首次系统量化该现象;(3)严格测试能有效揭示旧基准中普遍存在的“正确性幻觉”。本工作为软件工程领域大模型评估提供了可扩展、可诊断的新范式。代码、数据与结果详见 https://code2bench.github.io/。
原文摘要 · Abstract (English)
The evaluation of code-generating Large Language Models (LLMs) is fundamentally constrained by two intertwined challenges: a reliance on static, easily contaminated problem sources and the use of superficial, low-rigor testing. This paper introduces a new benchmark construction philosophy, Dual Scaling, designed to systematically address both limitations. Our approach involves continuously scaling the source of problems from dynamic, real-world code repositories and systematically scaling the rigor of tests via automated, high-coverage Property-Based Testing (PBT). We instantiate this philosophy in CODE2BENCH, an end-to-end framework that leverages Scope Graph analysis for principled dependency classification and a 100% branch coverage quality gate to ensure test suite integrity. Using this framework, we construct CODE2BENCH-2509, a new benchmark suite with native instances in both Python and Java. Our extensive evaluation of 10 state-of-the-art LLMs on CODE2BENCH-2509, powered by a novel "diagnostic fingerprint" visualization, yields three key insights: (1) models exhibit a fundamental performance gap, excelling at API application (Weakly Self-Contained tasks) but struggling with algorithmic synthesis (Self-Contained tasks); (2) a model's performance is profoundly shaped by the target language's ecosystem, a nuance we are the first to systematically quantify; and (3) our rigorous, scaled testing is critical in uncovering an "illusion of correctness" prevalent in simpler benchmarks. Our work presents a robust, scalable, and diagnostic paradigm for the next generation of LLM evaluation in software engineering. The code, data, and results are available at https://code2bench.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。