让大模型在推理时每一步都得分,提升数学解题能力
Prioritize the Process, Not Just the Outcome: Rewarding Latent Thought Trajectories Improves Reasoning in Looped Language Models
- 用全程奖励思维轨迹替代只看最终答案的强化学习方法
- 1.4B模型数学题准确率提5.8%,2.6B模型提10.9%
- 不依赖外部验证器,适用于各类推理任务
Looped Language Models(LoopLMs)在生成文本前进行多步隐式推理,在参数量较小的情况下优于传统大模型。然而,以往用强化学习优化推理效果的努力失败了——标准方法如组相对策略优化(GRPO)仅对最终隐状态赋分,与模型内部计算过程不匹配。为此,我们提出RLTT(Reward Latent Thought Trajectories)框架,将奖励分布到完整的隐式推理轨迹中。该方法无需外部验证器即可实现密集的轨迹级奖励分配,可无缝替换GRPO且开销极低。在相同训练与推理条件下,对Ouro-1.4B/2.6B-Thinking模型的大量实验显示,RLTT在挑战性数学推理基准上显著优于GRPO:1.4B模型在MATH-500、AIME24/26和BeyondAIME上平均准确率提升+5.8%,2.6B模型提升+10.9%。尽管仅在数学任务上训练,该方法仍有效迁移到非数学推理任务,证明了轨迹级奖励分配在循环语言模型强化学习中的有效性。代码已公开于https://github.com/jonwill8/RLTT.git。
原文摘要 · Abstract (English)
Looped Language Models (LoopLMs) perform multi-step latent reasoning prior to token generation and outperform conventional LLMs on reasoning benchmarks at smaller parameter budgets. However, attempts to further improve LoopLM reasoning with reinforcement learning have failed - standard objectives such as Group Relative Policy Optimization (GRPO) only assign credit to the final latent state, creating a fundamental mismatch with the model's internal computation. To resolve this, we introduce RLTT (Reward Latent Thought Trajectories), a reinforcement learning framework which distributes reward across the full latent reasoning trajectory. RLTT provides dense, trajectory-level credit assignment without relying on external verifiers and can directly replace GRPO with negligible overhead. Across extensive experiments with Ouro-1.4B/2.6B-Thinking under identical training and inference conditions, RLTT yields statistically significant improvements over GRPO on challenging mathematical reasoning benchmarks, improving mean accuracy over MATH-500, AIME24/26, and BeyondAIME by +5.8% on the 1.4B scale, and +10.9% on the 2.6B scale. Despite being trained exclusively on mathematics, RLTT also transfers effectively to non-mathematical reasoning benchmarks, demonstrating the effectiveness of trajectory-level credit assignment for reinforcement learning in LoopLMs. Code is available at https://github.com/jonwill8/RLTT.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。