arXiv:2502.06807cs.LGcs.AI2025-02被引 118

大模型通过强化学习在编程竞赛中表现卓越,通用模型已超越专用系统。

Competitive Programming with Large Reasoning Models

  • 用强化学习训练大模型,提升复杂编程与推理能力。
  • o3模型在无特殊策略下获IOI2024金牌,代码力媲美顶尖人类选手。
  • 通用模型优于手工设计的专用系统,适合追求通用智能的研究者。

我们展示,将强化学习应用于大型语言模型(LLMs)能显著提升其在复杂编码与推理任务中的表现。我们对比了两种通用推理模型——OpenAI o1 和 o3 的早期版本——与一个领域专用系统 o1-ioi,后者采用人工设计的推理策略,专为应对 2024 年国际信息学奥林匹克竞赛(IOI)而构建。我们在 IOI 2024 中以 o1-ioi 实时参赛,在使用人工测试时策略的情况下排名第 49 百分位;在放宽竞赛约束条件下,获得金牌。然而,当评估后续模型 o3 时,发现其无需任何手工设计的领域特定策略或宽松约束即可取得金牌。o3 在 2024 年 IOI 中获得金牌,并达到与精英人类选手相当的 Codeforces 评级。结果表明,相比依赖领域特定技术,扩大通用模型的强化学习规模,是通往推理类任务顶尖人工智能的更稳健路径。

原文摘要 · Abstract (English)

We show that reinforcement learning applied to large language models (LLMs) significantly boosts performance on complex coding and reasoning tasks. Additionally, we compare two general-purpose reasoning models - OpenAI o1 and an early checkpoint of o3 - with a domain-specific system, o1-ioi, which uses hand-engineered inference strategies designed for competing in the 2024 International Olympiad in Informatics (IOI). We competed live at IOI 2024 with o1-ioi and, using hand-crafted test-time strategies, placed in the 49th percentile. Under relaxed competition constraints, o1-ioi achieved a gold medal. However, when evaluating later models such as o3, we find that o3 achieves gold without hand-crafted domain-specific strategies or relaxed constraints. Our findings show that although specialized pipelines such as o1-ioi yield solid improvements, the scaled-up, general-purpose o3 model surpasses those results without relying on hand-crafted inference heuristics. Notably, o3 achieves a gold medal at the 2024 IOI and obtains a Codeforces rating on par with elite human competitors. Overall, these results indicate that scaling general-purpose reinforcement learning, rather than relying on domain-specific techniques, offers a robust path toward state-of-the-art AI in reasoning domains, such as competitive programming.

大模型推理增强编程竞赛强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。