为循环语言模型设计强化预训练，让中间推理更高效准确
LoopRPT: Reinforcement Pre-Training for Looped Language Models
- 将下一个词预测转为推理任务，用教师模型指导隐变量更新
- 在多个规模模型上提升每步表示质量，减少迭代次数同时保持精度
- 适合研究高效推理架构的学者，尤其关注隐式思维链的场景
循环语言模型（LoopLM）通过迭代隐变量计算来优化内部表征，是显式思维链（CoT）推理的有前途替代方案。然而，现有强化学习范式主要针对输出词元，与循环架构中隐式展开的推理存在结构不匹配。本文提出专为LoopLM设计的强化预训练框架LoopRPT，将下一个词预测重构为下一个词推理任务，利用指数移动平均教师参考和噪声隐变量轨迹，直接向隐变量步骤分配强化信号。该设定使强化学习能直接塑造中间表征，将有效推理压缩至更少迭代。我们在Ouro架构上实现了不同规模的LoopRPT实例。结果表明，LoopRPT持续提升每步表征质量，在准确率-计算量权衡中实现帕累托占优。尤其在困难词元上显著增益，表明其增强了早期推理能力，而非仅诱导过早退出。研究揭示强化预训练是学习高效隐变量推理的合理范式。
原文摘要 · Abstract (English)
Looped language models (LoopLMs) perform iterative latent computation to refine internal representations, offering a promising alternative to explicit chain-of-thought (CoT) reasoning. However, existing reinforcement learning (RL) paradigms primarily target output tokens, creating a structural mismatch with looped architectures whose reasoning unfolds implicitly. In this work, we propose LoopRPT, a reinforcement pre-training framework tailored for LoopLMs. By reframing next-token prediction as a next-token reasoning task, LoopRPT assigns reinforcement signals directly to latent steps using an EMA teacher reference and noisy latent rollouts. This formulation enables RL to directly shape intermediate representations, compressing effective reasoning into fewer iterations. We instantiate LoopRPT on the Ouro architecture across multiple model scales. Results demonstrate that LoopRPT consistently improves per-step representation quality, achieving Pareto dominance in accuracy-computation trade-offs. Notably, significant gains on hard tokens indicate that LoopRPT enhances early-stage reasoning rather than merely encouraging premature exits. Our findings highlight reinforcement pre-training as a principled paradigm for learning efficient latent reasoning in LoopLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。