多语言代码生成模型评估揭示了单指标排行榜掩盖的性能差异。
Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMs
- 在12种语言2707道题上执行测试，保留完整代码与运行结果。
- 最佳模型正确率仅23.64%，远低于人类57.2%的基准。
- 编译错误占非通过提交的63.25%，静态质量与功能正确性不一致。
代码生成模型通常用紧凑的执行基准和平均通过率进行比较，但这类汇总会掩盖性能在编程语言、问题类型和失败模式间的差异。我们对9个开源代码大模型在12种编程语言的2,707道LeetCode题目上进行了大规模、基于执行的评估，共产生325,343个问题-模型-语言任务，每个任务关联提示元数据、提取代码、LeetCode执行结果及静态分析信号。结果显示，当前开源模型距离人类接受基准仍有很大差距：最优模型Yi-Coder-9B-Chat的平均正确率为23.64%，而人类基准为57.2%。排名依赖于评估切片：Qwen2.5-Coder-14B-Instruct在难题和多样题覆盖上表现最强，而Gemma-2-27B-IT在全语言Lint通过率最高。失败分析表明，63.25%的未通过最佳提交因编译错误导致，说明许多失败发生在语义正确性验证之前。静态质量与功能正确性存在显著分歧。这些发现表明，多语言、可复现的评估能揭示单语言或单指标榜单所隐藏的权衡。
原文摘要 · Abstract (English)
Code generation models are typically compared using compact execution benchmarks and aggregate pass rates, but such summaries obscure how performance varies across programming languages, problem families, and failure modes. We present a large-scale, execution-grounded evaluation of 9 openly accessible LLMs specialized for coding on 2,707 free LeetCode problems across 12 programming languages. Our corpus contains 325,343 problem-model-language jobs, each linked to prompt metadata, extracted code, LeetCode execution outcomes, and static-analysis signals. The results show that current open models remain far from the human acceptance reference: the best model, Yi-Coder-9B-Chat, reaches 23.64% mean correctness, compared with a 57.2% human acceptance baseline. Rankings are also slice-dependent: Qwen2.5-Coder-14B-Instruct is strongest on hard problems and distinct-problem coverage, while Gemma-2-27B-IT achieves the highest all-language lint pass rate. Failure analysis shows that compile errors account for 63.25% of non-accepted best submissions, indicating that many failures occur before semantic correctness can be tested. Static quality further diverges from functional correctness. Together, these findings show that multilingual, artifact-preserving evaluation reveals tradeoffs hidden by single-language or single-metric leaderboards.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。