评测大模型代码生成的正确性、效率与质量，发现高正确率不等于高效好代码。
COMPASS: A Multi-Dimensional Benchmark for Evaluating Code Generation in Large Language Models
- 构建三维度评估框架，覆盖代码正确性、运行效率与可维护性。
- 基于39.3万真实提交数据，测试3款主流模型在50道编程题表现。
- 揭示当前模型虽正确率高，但代码效率和质量普遍不足，适合研发与评测人员参考。
现有代码生成评测主要关注功能正确性，忽略了真实编程中的算法效率与代码质量。我们提出COMPASS（COdility's Multi-dimensional Programming ASSessment），一个涵盖正确性、效率与质量三个维度的综合性评估框架。该框架包含来自真实Codility竞赛的50道编程题，并提供393,150份人类提交作为基准。不同于传统评测中将低效与最优解等同处理的做法，COMPASS使用行业标准分析工具系统评估运行时效率与代码质量。对Anthropic Claude Opus 4、Google Gemini 2.5 Pro和OpenAI O4-Mini-High三款领先推理增强模型的评估显示，高正确率模型并不一定生成高效或可维护的代码。这一结果凸显了超越正确性评估的重要性，有助于推动更可靠、可投入生产的代码生成系统研究。
原文摘要 · Abstract (English)
Current code generation benchmarks focus primarily on functional correctness while overlooking two critical aspects of real-world programming: algorithmic efficiency and code quality. We introduce COMPASS (COdility's Multi-dimensional Programming ASSessment), a comprehensive evaluation framework that assesses code generation across three dimensions: correctness, efficiency, and quality. COMPASS consists of 50 competitive programming problems from real Codility competitions, providing authentic human baselines from 393,150 submissions. Unlike existing benchmarks that treat algorithmically inefficient solutions identically to optimal ones provided they pass test cases, COMPASS systematically evaluates runtime efficiency and code quality using industry-standard analysis tools. Our evaluation of three leading reasoning-enhanced models, Anthropic Claude Opus 4, Google Gemini 2.5 Pro, and OpenAI O4-Mini-High, reveals that models achieving high correctness scores do not necessarily produce efficient algorithms or maintainable code. These findings highlight the importance of evaluating more than just correctness to truly understand the real-world capabilities of code generation models. COMPASS serves as a guiding framework, charting a path for future research toward AI systems that are robust, reliable, and ready for production use.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。