用国际编程竞赛题评估大模型推理能力，发现顶尖模型需多轮纠错才能发挥潜力。
ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests
- 基于真实国际编程竞赛题目构建评测集，模拟高难度竞赛环境。
- 提出新评估指标Refine@K，支持根据运行反馈迭代修复代码。
- 揭示当前大模型仍落后于人类顶尖选手，需多轮交互才能发挥潜力。
随着大语言模型在复杂编码与推理任务中的显著进步，现有基准如LiveCodeBench和CodeElo已无法充分评估其在真实竞赛环境下的编码能力。此外，传统评估指标Pass@K难以捕捉模型的反思能力。为此，我们提出ICPC-Eval——一个面向顶级编程竞赛场景的评测基准，包含11场全球范围内近期举办的ICPC竞赛中精心筛选的118道题目。该基准具有三大贡献：1）构建贴近真实竞赛的问题类型与难度分布；2）设计稳健的测试用例生成方法及本地评估工具包，实现高效精准的本地评估；3）引入有效测试时缩放评估指标Refine@K，支持基于执行反馈的代码迭代修正。实验表明，即使顶尖推理模型（如DeepSeek-R1）也需依赖多轮代码反馈才能完全释放其上下文推理潜能，相较非推理模型表现更优。尽管代码生成能力持续提升，当前模型仍显著落后于顶级人类参赛团队。基准已开源：https://github.com/RUCAIBox/Slow_Thinking_with_LLMs
原文摘要 · Abstract (English)
With the significant progress of large reasoning models in complex coding and reasoning tasks, existing benchmarks, like LiveCodeBench and CodeElo, are insufficient to evaluate the coding capabilities of large language models (LLMs) in real competition environments. Moreover, current evaluation metrics such as Pass@K fail to capture the reflective abilities of reasoning models. To address these challenges, we propose \textbf{ICPC-Eval}, a top-level competitive coding benchmark designed to probing the frontiers of LLM reasoning. ICPC-Eval includes 118 carefully curated problems from 11 recent ICPC contests held in various regions of the world, offering three key contributions: 1) A challenging realistic ICPC competition scenario, featuring a problem type and difficulty distribution consistent with actual contests. 2) A robust test case generation method and a corresponding local evaluation toolkit, enabling efficient and accurate local evaluation. 3) An effective test-time scaling evaluation metric, Refine@K, which allows iterative repair of solutions based on execution feedback. The results underscore the significant challenge in evaluating complex reasoning abilities: top-tier reasoning models like DeepSeek-R1 often rely on multi-turn code feedback to fully unlock their in-context reasoning potential when compared to non-reasoning counterparts. Furthermore, despite recent advancements in code generation, these models still lag behind top-performing human teams. We release the benchmark at: https://github.com/RUCAIBox/Slow_Thinking_with_LLMs
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。