arXiv:2509.25827cs.CLcs.AI2025-09中稿 · ICLR被引 4

解决大模型推理过长问题,效率提升超50%且不降性能

Overthinking Reduction with Decoupled Rewards and Curriculum Data Scheduling

  • 分离令牌级奖励机制,精准识别并惩罚冗余推理步骤
  • 在7个基准上减少超50%推理令牌,性能不变或提升
  • 适合追求高效推理的AI研发与部署人员

尽管无评判器强化学习与可验证奖励(RLVR)训练的大规模推理模型代表当前前沿,但其实际应用受限于“过度思考”问题——模型生成过长推理路径却无性能提升。现有通过惩罚长度的方法常失效,因轨迹级奖励与令牌级优化存在根本性错位。本文基于理论发现,指出当前长度奖励的两大缺陷:(1) 错误惩罚必要探索性令牌;(2) 不慎奖励部分冗余。提出新框架DECS,包含两项创新:(i) 首个分离式令牌级奖励机制,精准区分并惩罚冗余令牌;(ii) 新型课程化批次调度策略,平衡效率与有效性。实验显示,DECS在7个基准上推理令牌数减少超过50%,同时性能保持甚至提升。结果明确表明,可在不损害模型推理能力的前提下实现显著效率提升。代码已开源。

原文摘要 · Abstract (English)

While large reasoning models trained with critic-free reinforcement learning and verifiable rewards (RLVR) represent the state-of-the-art, their practical utility is hampered by ``overthinking'', a critical issue where models generate excessively long reasoning paths without any performance benefit. Existing solutions that penalize length often fail, inducing performance degradation due to a fundamental misalignment between trajectory-level rewards and token-level optimization. In this work, we introduce a novel framework, DECS, built on our theoretical discovery of two previously unaddressed flaws in current length rewards: (1) the erroneous penalization of essential exploratory tokens and (2) the inadvertent rewarding of partial redundancy. Our framework's innovations include (i) a first-of-its-kind decoupled token-level reward mechanism that surgically distinguishes and penalizes redundant tokens, and (ii) a novel curriculum batch scheduling strategy to master the efficiency-efficacy equilibrium. Experimental results show DECS can achieve a dramatic reduction in reasoning tokens by over 50\% across seven benchmarks while simultaneously maintaining or even improving performance. It demonstrates conclusively that substantial gains in reasoning efficiency can be achieved without compromising a model's underlying reasoning power. Code is available at https://github.com/pixas/DECS.

推理优化强化学习效率提升

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。