用折扣强化学习让大模型推理更短更准，节省算力。
Learning to Reason Efficiently with Discounted Reinforcement Learning
- 用折扣强化学习惩罚推理token，鼓励简洁思考
- 实验显示推理长度减少30%以上，准确率不变
- 适合追求高效推理的部署场景
大型推理模型（LRMs）常消耗过多文本标记，增加计算成本和延迟。在目标达成的序列决策问题中，我们希望快速抵达目标，而大型模型的推理过程可从这一视角理解。本文挑战了‘更长的回答更准确’的假设。通过使用折扣强化学习框架（可解释为微小的标记成本）惩罚推理标记，并在受限策略类中分析Blackwell最优性，我们促使模型在保持准确性的前提下生成更简洁的推理过程，类似于在随机最短路径问题中偏好更短的成功轨迹。实验验证了理论结果：该方法显著缩短思维链长度，同时维持原有准确率。
原文摘要 · Abstract (English)
Large reasoning models (LRMs) often consume excessive tokens, inflating computational cost and latency. More broadly, in goal reaching sequential decision problems we often want to reach the goal quickly, and LRM reasoning can be viewed through this lens. We challenge the assumption that longer responses improve accuracy. By penalizing reasoning tokens using a discounted reinforcement learning setup (interpretable as a small token cost) and analyzing Blackwell optimality in restricted policy classes, we encourage concise yet accurate reasoning, analogous to preferring shorter successful trajectories in a stochastic shortest path problem. Experiments confirm our theoretical results that this approach shortens chains of thought while preserving accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。