用奥运金牌选手标准评估大模型编程能力,发现其仍远不如人类专家。
LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?
- 由奥赛金牌选手标注代码题,逐行分析模型错误。
- 无工具时,模型中等题通过率仅53%,难题为0%。
- 模型擅长实现但缺乏算法推理,适合研究代码生成缺陷。
有报告称大语言模型(LLMs)在编程竞赛中已超越顶尖人类。基于国际算法竞赛金牌得主的知识,我们重新审视这一说法,分析模型与人类专家的差异及现存局限。我们提出LiveCodeBench Pro基准,包含持续更新的Codeforces、ICPC和IOI题目,以降低数据污染风险。一支奥赛金牌团队对每道题进行算法分类,并对模型生成的失败提交进行逐行分析。利用该数据与基准,我们发现前沿模型仍有显著不足:在无外部工具情况下,最佳模型在中等难度问题上仅达53% pass@1,难题为0%;模型在实现密集型任务上表现良好,但在复杂算法推理与边界情况分析上表现不佳,常给出自信却错误的解释。高性能主要源于实现精度和工具增强,而非优越推理能力。LiveCodeBench Pro揭示了模型与人类大师级水平间的巨大差距,同时提供细粒度诊断,推动未来代码导向型模型推理的改进。
原文摘要 · Abstract (English)
Recent reports claim that large language models (LLMs) now outperform elite humans in competitive programming. Drawing on knowledge from a group of medalists in international algorithmic contests, we revisit this claim, examining how LLMs differ from human experts and where limitations still remain. We introduce LiveCodeBench Pro, a benchmark composed of problems from Codeforces, ICPC, and IOI that are continuously updated to reduce the likelihood of data contamination. A team of Olympiad medalists annotates every problem for algorithmic categories and conducts a line-by-line analysis of failed model-generated submissions. Using this new data and benchmark, we find that frontier models still have significant limitations: without external tools, the best model achieves only 53% pass@1 on medium-difficulty problems and 0% on hard problems, domains where expert humans still excel. We also find that LLMs succeed at implementation-heavy problems but struggle with nuanced algorithmic reasoning and complex case analysis, often generating confidently incorrect justifications. High performance appears largely driven by implementation precision and tool augmentation, not superior reasoning. LiveCodeBench Pro thus highlights the significant gap to human grandmaster levels, while offering fine-grained diagnostics to steer future improvements in code-centric LLM reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。