通过强化学习优化推理路径，减少大模型响应长度。
Shorten After You're Right: Lazy Length Penalties for Reasoning RL
- 在强化学习中设计懒惰长度惩罚，直接缩短推理过程。
- 逻辑题减少40%步骤，性能提升14%；数学题减少33%步骤，性能不变。
- 无需额外训练阶段，适合追求高效推理的部署场景。
大型推理模型如OpenAI o1和DeepSeek R1在推理任务上表现卓越，但常伴随冗长的推理路径，导致显著的内存与时间开销。现有方法主要通过引入额外训练数据和阶段来压缩路径。本文提出三种关键奖励机制，直接嵌入大模型的强化学习过程中，无需额外训练阶段即可减少响应长度。在四个设置下的实验表明，该方法显著降低响应长度，同时保持甚至提升性能。具体而言，在逻辑推理任务中，平均步骤减少40%，性能提升14%；在数学问题上，平均步骤减少33%，性能保持不变。
原文摘要 · Abstract (English)
Large reasoning models, such as OpenAI o1 or DeepSeek R1, have demonstrated remarkable performance on reasoning tasks but often incur a long reasoning path with significant memory and time costs. Existing methods primarily aim to shorten reasoning paths by introducing additional training data and stages. In this paper, we propose three critical reward designs integrated directly into the reinforcement learning process of large reasoning models, which reduce the response length without extra training stages. Experiments on four settings show that our method significantly decreases response length while maintaining or even improving performance. Specifically, in a logic reasoning setting, we achieve a 40% reduction in response length averaged by steps alongside a 14% gain in performance. For math problems, we reduce response length averaged by steps by 33% while preserving performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。