用人类水平的爱洛评分，首次评估大模型代码生成能力。
CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings
- 基于CodeForces平台构建真实竞赛题集，支持自动评测与判题。
- o1-mini和QwQ-32B-Preview分别达1578和1261分，远超其他模型。
- 适合关注大模型编程能力、竞赛级代码生成的研究者使用。
随着大语言模型（LLMs）代码推理能力提升及推理模型如OpenAI o1、o3的突破，亟需更具挑战性与全面性的基准来评估其高水平编程能力。现有基准如LiveCodeBench和USACO因缺乏私有测试用例、无特殊判题支持、执行环境不一致而存在局限。为此，我们提出CodeElo——首个标准化竞赛级代码生成基准，基于官方CodeForces平台构建，涵盖最近六个月比赛题目，包含比赛分组、难度评级与算法标签等详细信息。引入直接提交至平台的判题机制，并建立可靠且可比人类的爱洛评分系统，方差更低。在该基准上，首次为30个主流开源与专有模型提供爱洛评分。结果表明，o1-mini与QwQ-32B-Preview表现显著，分别获得1578与1261分；其余模型连最简单题目都难以应对，排名低于所有人类参与者中前25%。通过算法维度分析与C++/Python对比实验，揭示性能差异，为后续研究提供方向。
原文摘要 · Abstract (English)
With the increasing code reasoning capabilities of existing large language models (LLMs) and breakthroughs in reasoning models like OpenAI o1 and o3, there is a growing need to develop more challenging and comprehensive benchmarks that effectively test their sophisticated competition-level coding abilities. Existing benchmarks, like LiveCodeBench and USACO, fall short due to the unavailability of private test cases, lack of support for special judges, and misaligned execution environments. To bridge this gap, we introduce CodeElo, a standardized competition-level code generation benchmark that effectively addresses all these challenges for the first time. CodeElo benchmark is mainly based on the official CodeForces platform and tries to align with the platform as much as possible. We compile the recent six months of contest problems on CodeForces with detailed information such as contest divisions, problem difficulty ratings, and problem algorithm tags. We introduce a unique judging method in which problems are submitted directly to the platform and develop a reliable Elo rating calculation system that aligns with the platform and is comparable with human participants but has lower variance. By testing on our CodeElo, we provide the Elo ratings of 30 existing popular open-source and 3 proprietary LLMs for the first time. The results show that o1-mini and QwQ-32B-Preview stand out significantly, achieving Elo ratings of 1578 and 1261, respectively, while other models struggle even with the easiest problems, placing in the lowest 25 percent among all human participants. Detailed analysis experiments are also conducted to provide insights into performance across algorithms and comparisons between using C++ and Python, which can suggest directions for future studies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。