构建多粒度多语言代码生成评估框架,揭示模型在不同代码层级的表现差异。
M2G-Eval: Enhancing and Evaluating Multi-granularity Multilingual Code Generation
- 按类、函数、块、行四个层级设计多粒度评估框架
- 覆盖18种语言,包含1.7万+训练任务和1286个人工标注测试集
- 发现代码层级越复杂,跨语言性能差距越大,且编程概念可迁移
代码大模型的快速发展激发了对其生成能力系统评估的研究兴趣,但现有基准大多仅在单一结构粒度上评估,且局限于少数编程语言,难以揭示不同代码范围和多语言场景下的细微能力差异。我们提出M2G-Eval,一个面向代码生成的多粒度、多语言评估框架,涵盖类、函数、块、行四个层级。该框架覆盖18种编程语言,包含17,000多个训练任务和1,286个经过人工标注、污染控制的测试实例。我们通过监督微调与组相对策略优化,基于Qwen3-8B训练出M2G-Eval-Coder模型。对30个模型(28个先进大模型及我们的两个M2G-Eval-Coder变体)的评估揭示三大发现:(1) 明显的难度层次,行级任务最易,类级最难;(2) 随着任务复杂度增加,全语言与部分语言支持模型间的性能差距扩大;(3) 强跨语言相关性,表明模型能学习可迁移的编程概念。M2G-Eval可实现代码生成能力的细粒度诊断,并凸显合成复杂长代码的持续挑战。
原文摘要 · Abstract (English)
The rapid advancement of code large language models (LLMs) has sparked significant research interest in systematically evaluating their code generation capabilities, yet existing benchmarks predominantly assess models at a single structural granularity and focus on limited programming languages, obscuring fine-grained capability variations across different code scopes and multilingual scenarios. We introduce M2G-Eval, a multi-granularity, multilingual framework for evaluating code generation in large language models (LLMs) across four levels: Class, Function, Block, and Line. Spanning 18 programming languages, M2G-Eval includes 17K+ training tasks and 1,286 human-annotated, contamination-controlled test instances. We develop M2G-Eval-Coder models by training Qwen3-8B with supervised fine-tuning and Group Relative Policy Optimization. Evaluating 30 models (28 state-of-the-art LLMs plus our two M2G-Eval-Coder variants) reveals three main findings: (1) an apparent difficulty hierarchy, with Line-level tasks easiest and Class-level most challenging; (2) widening performance gaps between full- and partial-granularity languages as task complexity increases; and (3) strong cross-language correlations, suggesting that models learn transferable programming concepts. M2G-Eval enables fine-grained diagnosis of code generation capabilities and highlights persistent challenges in synthesizing complex, long-form code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。