用分段加权惩罚缓解大模型推理过长问题，提升效率不丢性能。
Mitigating Overthinking through Reasoning Shaping
- 按推理片段分组，动态加权惩罚，控制冗余思考
- 减少40%以上推理令牌数，难题表现提升明显
- 适合追求高效推理的模型部署与训练场景
基于验证器奖励的强化学习（RLVR）驱动的大推理模型虽具强大解题能力，但常出现过度思考：冗长、散漫的推理过程显著增加计算开销。以往的令牌级惩罚设计虽降低令牌消耗，却往往损害模型性能，根源在于监督粒度过于粗糙。本文认为监督粒度对效率与准确率的平衡至关重要，提出分组相对片段惩罚（GRSP），一种基于步骤的正则化方法。初步分析表明，推理片段与令牌消耗及模型性能强相关，因此设计了跨片段簇的长度感知加权机制。大量实验显示，GRSP在不严重牺牲准确率的前提下实现更优的令牌效率，尤其在复杂问题上优势显著；同时稳定了强化学习训练，并可有效扩展至不同规模模型。
原文摘要 · Abstract (English)
Large reasoning models (LRMs) boosted by Reinforcement Learning from Verifier Reward (RLVR) have shown great power in problem solving, yet they often cause overthinking: excessive, meandering reasoning that inflates computational cost. Prior designs of penalization in RLVR manage to reduce token consumption while often harming model performance, which arises from the oversimplicity of token-level supervision. In this paper, we argue that the granularity of supervision plays a crucial role in balancing efficiency and accuracy, and propose Group Relative Segment Penalization (GRSP), a step-level method to regularize reasoning. Since preliminary analyses show that reasoning segments are strongly correlated with token consumption and model performance, we design a length-aware weighting mechanism across segment clusters. Extensive experiments demonstrate that GRSP achieves superior token efficiency without heavily compromising accuracy, especially the advantages with harder problems. Moreover, GRSP stabilizes RL training and scales effectively across model sizes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。