arXiv:2604.26923cs.SEcs.CL2026-04中稿 · AIware 2026被引 3

构建首个跨领域类级代码生成基准,揭示大模型在此任务上的显著能力差距。

ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generation

论文配图:ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generation
图 1 · 摘自论文原文
  • 通过自动化流程生成300个跨领域类级代码任务,融合真实开源代码。
  • 最强模型仅达45.6%通过率,模型间差距高达17.7个百分点。
  • 适合评估大模型在复杂结构代码生成中的协同与依赖处理能力。

大型语言模型在函数级代码生成和项目级代码修改上已取得优异表现,但介于二者之间的组合式代码生成——即从规范中构建完整、内部结构化的类——仍缺乏有效评估。现有评测或局限于孤立函数,或依赖人工构建的类级任务,难以扩展且易受数据污染。我们提出ClassEval-Pro,一个涵盖11个领域的300个类级任务基准,通过自动化三阶段流程构建:复杂度增强、跨域类组合、整合2025年1月后贡献的真实GitHub代码。每个任务经由大模型判别器集合验证,并需通过测试套件且覆盖率超90%。我们在五种前沿大模型上评估五种生成策略。最佳模型仅达45.6%的Pass@1率,强弱模型间差距达17.7个百分点,证实该基准具有强区分力。策略选择与模型能力强烈交互:自底向上的结构化方法可使弱模型提升9.4个百分点,而组合生成降至最低1.3%。对500个手动标注失败案例的分析显示,逻辑错误(56.2%)与依赖错误(38.0%)主导,表明跨方法协调是核心瓶颈。

原文摘要 · Abstract (English)

LLMs have achieved strong results on both function-level code synthesis and repository-level code modification, yet a capability that falls between these two extremes -- compositional code creation, i.e., building a complete, internally structured class from a specification -- remains underserved. Current evaluations are either confined to isolated functions or rely on manually curated class-level tasks that are expensive to scale and increasingly susceptible to data contamination. We introduce ClassEval-Pro, a benchmark of 300 class-level tasks spanning 11 domains, constructed through an automated three-stage pipeline that combines complexity enhancement, cross-domain class composition, and integration of real-world GitHub code contributed after January 2025. Every task is validated by an LLM Judge Ensemble and must pass test suites with over 90% line coverage. We evaluate five frontier LLMs under five generation strategies. The best model achieves only 45.6% class-level Pass@1, with a 17.7-point gap between the strongest and weakest models, confirming the benchmark's discriminative power. Strategy choice strongly interacts with model capability: structured approaches such as bottom-up improve weaker models by up to 9.4 percentage points, while compositional generation collapses to as low as 1.3%. Error analysis over 500 manually annotated failures reveals that logic errors (56.2%) and dependency errors (38.0%) dominate, identifying cross-method coordination as the core bottleneck.

代码生成大模型评估跨领域类级生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。