让大模型自动判断何时停止推理，既省算力又不丢准确率。
Step-GRPO: Internalizing Dynamic Early Exit for Efficient Reasoning
- 用语义步骤替代token数优化，让模型学会识别有效推理阶段。
- 在Qwen3-8B上减少32%的计算量，且准确率不下降。
- 适合需要高效推理的部署场景，尤其对资源受限设备友好。
采用长思维链的大模型在解题上表现优异，但会因冗余检查浪费计算资源。抑制过度思考颇具挑战：训练时的长度惩罚可能损害模型能力，而推理时的提前退出则增加系统开销。为此，我们提出Step-GRPO——一种新型后训练框架，将动态提前退出能力直接内化到模型中。Step-GRPO通过语言标记构建推理结构，将优化目标从原始token转向语义步骤。我们引入动态截断回放机制，在探索过程中暴露模型于简洁且高置信度的推理轨迹，并结合步数感知相对奖励，基于群体基准动态惩罚冗余。在三种模型规模、多个基准上的实验表明，Step-GRPO实现了更优的准确性与效率权衡。在Qwen3-8B上，相比原模型，本方法将令牌消耗降低32.0%，且未出现传统长度惩罚方法中的准确率下降现象。
原文摘要 · Abstract (English)
Large reasoning models that use long chain-of-thought excel at problem-solving yet waste compute on redundant checks. Curbing this overthinking is hard: training-time length penalties can cripple ability, while inference-time early-exit adds system overhead. To bridge this gap, we propose Step-GRPO, a novel post-training framework that internalizes dynamic early-exit capabilities directly into the model. Step-GRPO shifts the optimization objective from raw tokens to semantic steps by utilizing linguistic markers to structure reasoning. We introduce a Dynamic Truncated Rollout mechanism that exposes the model to concise high-confidence trajectories during exploration, synergized with a Step-Aware Relative Reward that dynamically penalizes redundancy based on group-level baselines. Extensive experiments across three model sizes on diverse benchmarks demonstrate that Step-GRPO achieves a superior accuracy-efficiency trade-off. On Qwen3-8B, our method reduces token consumption by 32.0\% compared to the vanilla model while avoiding the accuracy degradation observed in traditional length-penalty methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。