通过预测未来标记引导循环变压器，提升推理能力并避免重复计算。
LoopMTP: A looped transformer guided by latent multi-token prediction
- 用潜在空间对齐实现跨循环的未来标记预测，提供前进式监督。
- 在15次循环下训练稳定，平均准确率提升8.1%（相对）。
- 适合需要高效推理且参数受限的长序列任务场景。
循环变压器作为深度扩展的参数高效替代方案，通过在T次迭代中复用同一层堆栈，在固定参数量下获得类似大模型的有效深度与推理能力。然而现有方法存在潜在过度思考和计算无差别问题，主要因中间表示在循环间缺乏指导。多标记预测（MTP）恰好为循环提供了密集、前瞻性的监督。我们提出 extsc{LoopMTP}，通过隐状态与未来标记嵌入之间的软对齐，使循环t的隐藏状态与t步后的标记建立对应关系；同时引入轻量门控机制保留有用信息。实验表明， extsc{LoopMTP}相较非循环基线平均准确率提升最高达8.1%（相对），且支持长达15次循环的稳定训练。
原文摘要 · Abstract (English)
Looped transformers have emerged as a parameter-efficient alternative to scaling depth for strong reasoning. By reusing one stack of layers across $T$ iterations, they attain the effective depth and reasoning capabilities of larger models at a fixed parameter count. Yet existing approaches suffer from latent overthinking and undifferentiated computation, largely because intermediate representations receive no guidance across loops. Multi-token prediction (MTP) supplies exactly the dense, forward-looking supervision the loop is missing. We propose \textsc{LoopMTP}, which links the two through a structural correspondence in latent space: a model that loops $T$ times can anticipate $T$ future tokens. \textsc{LoopMTP} realizes this by softly aligning the hidden state of loop $t$ with the embedding of the token $t$ steps ahead, while a lightweight gate preserves useful information across iterations. \textsc{LoopMTP} improves average accuracy by up to 8.1\% (relative) over the non-looped baseline, with training remaining stable for up to 15 loops.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。