提出环形Transformer残差缩放新规则,提升训练稳定性与模型迁移性。
On the Residual Scaling of Looped Transformers: Stability and Transferability

- 发现环形结构需用1/N而非1/√N作为残差缩放系数
- 实验表明1/N缩放在不同循环次数下均更优,损失更低
- 新规则支持跨规模超参数直接迁移,无需重新调参
环形(权值共享)Transformer通过重复使用同一残差块N次来增加有效深度而不增加参数量。已有深度扩展分析建议采用ε = 1/√L的缩放方式,但本文指出这对环形结构不足:权值共享导致各迭代间残差更新相关,必须采用更强的ε = 1/N缩放。对于包含L个独立层并循环N次的多层块结构,推导出因子化参数形式ε = λ/(N√L),其中1/N控制层内循环相关性,1/√L控制层间方差。关键结论是:最优学习率仅依赖于独立层数L,与循环次数N无关,实现从小规模到大规模模型的超参数直接迁移。在环形Transformer上的实验验证,1/N缩放显著提升可训练性,且在不同循环数下均优于1/√N缩放。
原文摘要 · Abstract (English)
Looped (weight-tied) Transformers apply a shared residual block $N$ times ($h \leftarrow h + \varepsilon\,f(h)$, same $f$ at each step), increasing effective depth without adding parameters. Prior depth-scaling analyses prescribe $\varepsilon = 1/\!\sqrt{L}$ for depth-$L$ residual networks. We show that this is insufficient for looped architectures: weight sharing makes residual updates correlated across iterations, requiring the stronger scaling $\varepsilon = 1/N$. For multi-layer blocks ($L$ unique layers looped $N$ times), we derive a factored parameterization $\varepsilon = λ/(N\!\sqrt{L})$ that separates the two sources of growth: $1/N$ controls the within-layer loop correlation, and $1/\!\sqrt{L}$ controls the across-layer variance. A key consequence is that the optimal learning rate depends only on the number of unique layers $L$, not on the loop count $N$, enabling direct hyperparameter transfer from small to large $N$ without retuning. Experiments on looped Transformers confirm that $1/N$ scaling improves trainability and yields better loss than $1/\!\sqrt{N}$ scaling across loop counts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。