arXiv:2607.13491cs.LGcs.AI2026-07被引 1

提出DeepLoop,解决循环变压器深度扩展时的残差缩放问题。

DeepLoop: Depth Scaling for Looped Transformers

  • 通过控制访问对齐系数,重新设计残差缩放规则以适配循环计算
  • 在GPT-2小和中等规模模型上,激活循环深度后验证损失下降,下游准确率提升
  • 适合研究循环结构或深度扩展的Transformer架构设计者

循环Transformer通过重复使用少量物理块实现序列计算的深度扩展,避免参数存储增加。这种重用改变了残差缩放机制:在未绑定的Transformer中,每个残差分支独立接收并应用参数更新;而在循环Transformer中,单一共享更新会聚合多次访问的梯度,并在下一次线性前向传递中被读回。我们通过一阶微扰界形式化了这一绑定深度效应,其受访问对齐系数κ_R控制。当访问不相关时,该界恢复DeepNorm指数;但在保守对齐情形下,随着循环次数增加而固定物理深度,指数需从1/4增至1/2。由此提出的DeepLoop方法保留Post-LN DeepNorm结构,设定α=(2N)^{1/2},β=(8N)^{-1/2},其中N为展开深度。在GPT风格的循环语言模型(GPT-2 small、GPT-2 medium)上,DeepLoop在无物理块重访时表现中性,一旦激活循环深度即改善验证损失与下游准确率。结果表明,稳定循环深度需要考虑参数访问次数而非名义层数的残差缩放规则。

原文摘要 · Abstract (English)

Looped Transformers scale sequential computation by applying a compact stack of physical blocks for multiple rounds, increasing unrolled depth without increasing stored parameters. This reuse changes the residual-scaling problem: in an untied Transformer, each residual branch receives and applies its own parameter update, whereas in a looped Transformer one shared update aggregates gradients from repeated visits and is read back by those same visits in the next linearized forward pass. We formalize this tied-depth effect through a first-order perturbation bound controlled by a visit-alignment coefficient $κ_R$. The bound recovers the DeepNorm exponent when visits decorrelate, but in the conservative aligned regime it requires the exponent to increase from $1/4$ to $1/2$ as loop count grows at fixed physical depth. The resulting method, \textbf{DeepLoop}, keeps the Post-LN DeepNorm architecture and sets $α=(2N)^{1/2}$ and $β=(8N)^{-1/2}$ for unrolled depth $N$. On GPT-style looped language models at GPT-2 small and GPT-2 medium scale, DeepLoop is neutral when no physical block is revisited and improves validation loss and downstream accuracy once recurrent depth is activated. These results show that stable recurrent depth requires residual scaling rules that account for parameter visits, not only nominal layer count.

Transformer循环结构深度扩展残差缩放

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。