ProBench评测大模型编程推理能力，聚焦竞赛级题目真实表现。
ProBench: Benchmarking Large Language Models in Competitive Programming
- 构建基于竞赛平台的动态评测基准，确保结果真实公平。
- QwQ-32B-Preview以20.93分领先，专用推理训练模型优势显著。
- 揭示算法适配与推理深度是提升编程能力的关键瓶颈。
随着OpenAI-o3和DeepSeek-R1等推理型大模型的出现，大语言模型进入新发展阶段。然而，现有代码评估基准逐渐难以衡量先进模型在代码推理方面的能力。为填补高水平代码推理评估的空白，我们提出ProBench，用于在竞赛编程场景下评测大模型，灵感来自国际大学生程序设计竞赛。ProBench从Codeforces、Luogu和Nowcoder平台（2024年7月至12月）收集了大量竞赛编程题，通过在线提交获取真实测试结果，确保评估的公平性与准确性。我们建立统一的问题属性体系，包括难度分级与算法标签。基于精心收集与标注的数据，系统评估了9个最新大模型在多个维度的表现，涵盖思维链分析、错误类型诊断与推理深度评估。实验结果表明，QwQ-32B-Preview以20.93分位居第一，优于DeepSeek-V3的16.38分，说明经过专门推理任务训练的模型在编程任务中显著超越通用模型（即使规模更大）。进一步分析揭示算法适配性与推理充分性是当前关键提升方向，为推理模型未来研发提供重要参考。
原文摘要 · Abstract (English)
With reasoning language models such as OpenAI-o3 and DeepSeek-R1 emerging, large language models (LLMs) have entered a new phase of development. However, existing benchmarks for coding evaluation are gradually inadequate to assess the capability of advanced LLMs in code reasoning. To bridge the gap for high-level code reasoning assessment, we propose ProBench to benchmark LLMs in competitive programming, drawing inspiration from the International Collegiate Programming Contest. ProBench collects a comprehensive set of competitive programming problems from Codeforces, Luogu, and Nowcoder platforms during the period from July to December 2024, obtaining real test results through online submissions to ensure the fairness and accuracy of the evaluation. We establish a unified problem attribute system, including difficulty grading and algorithm tagging. With carefully collected and annotated data in ProBench, we systematically assess 9 latest LLMs in competitive programming across multiple dimensions, including thought chain analysis, error type diagnosis, and reasoning depth evaluation. Experimental results show that QwQ-32B-Preview achieves the best score of 20.93 followed by DeepSeek-V3 with a score of 16.38, suggesting that models trained with specialized reasoning tasks significantly outperform general-purpose models (even larger than reasoning-oriented models) in programming. Further analysis also reveals key areas for programming capability enhancement, e.g., algorithm adaptability and reasoning sufficiency, providing important insights for the future development of reasoning models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。