arXiv:2608.18230cs.LG2026-08

只循环特定模块,用更少计算实现更强推理能力

Allocating Recurrent Compute in Looped Language Models

论文配图:Allocating Recurrent Compute in Looped Language Models
图 1 · 摘自论文原文
  • 仅重复门控混频器,不重复全连接层,节省计算
  • 在1500万参数下,核心任务性能超越全层循环41.5%以上
  • 适合追求高效推理的模型优化与资源受限部署

循环语言模型通过重复共享计算提升推理与知识操作能力。现有系统通常重复整个层堆栈,但混频器与密集前馈网络(FFN)执行不同操作且成本各异。本文聚焦更小问题:什么该循环?将循环视为状态更新的重复组合,提出当新跨位置影响方向在任务读出时仍可观察时,应用才具价值。迭代传输秩(ITR)描述累积影响轨迹;边际ITR表示每次应用带来的非冗余影响。据此提出MixerLoop:仅重复每个门控DeltaNet混频器,而全连接层仅执行一次。在1500万和1100万参数下,对比无循环与全块循环,在相同数据、初始化与架构条件下进行测试。有限上下文干预实验表明,后期混频器应用在最终语言模型读出处产生显著、非微弱且有益的变化。MixerLoop在1500万参数下超越全循环模型,且在1100万参数下仍保持41.5%的性能提升,同时将循环骨干投影浮点运算量减少45.9%。结果表明,无需重复执行密集前馈网络即可保留循环深度的优势。

原文摘要 · Abstract (English)

Looped language models improve reasoning and knowledge manipulation by applying shared computation repeatedly. Existing systems usually repeat an entire layer stack, although a mixer and a dense feed-forward network (FFN) perform different operations and have different costs. We ask a narrower question: what should loop? We view recurrence as repeated composition of a state update and argue that an application is valuable when it exposes a new cross-position influence direction that remains observable at the task readout. Iterative Transport Rank (ITR) describes the cumulative influence trajectory; marginal ITR describes the nonredundant influence contributed by successive applications. This view motivates MixerLoop, which repeats each Gated DeltaNet mixer while applying its dense FFN once. We compare MixerLoop with no recurrence and full-block recurrence at 15M and 110M parameters under the same data, initialization, and architecture. A finite context-off intervention tests whether later mixer applications produce distinct, non-negligible, and beneficial changes at the final language-model readout. MixerLoop surpasses FullLoop on aggregate CORE at 15M and retains 41.5% of its CORE improvement at 110M while reducing recurrent-backbone projection FLOPs by 45.9%. These results show that the benefits of recurrent depth can be retained without repeatedly executing the dense FFN.

语言模型循环计算效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。