arXiv:2505.21178cs.CL2025-05被引 17

让大模型先学会走再跑,用强化学习生成更简洁的推理过程。

Walk Before You Run! Concise LLM Reasoning via Reinforcement Learning

  • 分两阶段强化学习:先练推理能力,再约束输出长度。
  • 在多个数学竞赛数据集上超越现有最强模型,且无需复杂训练范式。
  • 适合需要高效、简洁推理结果的研究者与应用落地场景。

随着测试时扩展成为大语言模型发展的关键前沿,当前先进的后训练方法越来越关注延长链式思维(CoT)响应长度,以提升推理能力并达到DeepSeek R1水平。然而,最新研究揭示了先进推理模型普遍存在过度思考现象,表现为长CoT响应中的冗余或重复思维模式。为此,本文提出一种简单而有效的两阶段强化学习框架ConciseR,实现大模型的简洁推理。第一阶段通过更多训练步数,利用带剪裁上限和动态采样组件的组相对策略优化(GRPO++),激励模型推理能力;第二阶段使用较少训练步数,通过长度感知的组相对策略优化(L-GRPO),明确约束简洁性并提升效率。重要的是,ConciseR仅在样本所有推演路径均正确后才优化响应长度,遵循‘先会走再跑’原则。大量实验表明,ConciseR生成的CoT推理响应更简洁,在AIME 2024、MATH-500、AMC 2023、Minerva和奥数基准测试中全面优于近期最先进推理模型,且采用零强化学习范式。

原文摘要 · Abstract (English)

As test-time scaling becomes a pivotal research frontier in Large Language Models (LLMs) development, contemporary and advanced post-training methodologies increasingly focus on extending the generation length of long Chain-of-Thought (CoT) responses to enhance reasoning capabilities toward DeepSeek R1-like performance. However, recent studies reveal a persistent overthinking phenomenon in state-of-the-art reasoning models, manifesting as excessive redundancy or repetitive thinking patterns in long CoT responses. To address this issue, in this paper, we propose a simple yet effective two-stage reinforcement learning framework for achieving concise reasoning in LLMs, named ConciseR. Specifically, the first stage, using more training steps, aims to incentivize the model's reasoning capabilities via Group Relative Policy Optimization with clip-higher and dynamic sampling components (GRPO++), and the second stage, using fewer training steps, explicitly enforces conciseness and improves efficiency via Length-aware Group Relative Policy Optimization (L-GRPO). Significantly, ConciseR only optimizes response length once all rollouts of a sample are correct, following the "walk before you run" principle. Extensive experimental results demonstrate that our ConciseR model, which generates more concise CoT reasoning responses, outperforms recent state-of-the-art reasoning models with zero RL paradigm across AIME 2024, MATH-500, AMC 2023, Minerva, and Olympiad benchmarks.

大模型推理强化学习简洁生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。