LayerNorm在循环Transformer中隐式控制增益，提升稳定性。
LayerNorm as Implicit Gain Control in Looped Transformers
- LayerNorm通过耦合局部Lipschitz常数与激活尺度，实现隐式增益控制。
- 即使算子范数大于1，系统在固定点仍渐近收缩，稳定依赖谱裕度。
- 适合研究循环神经网络稳定性和训练机制的学者阅读。
在预LayerNorm的循环Transformer中，递归块内的LayerNorm充当隐式增益控制器：它将块的局部Lipschitz常数与激活尺度反向耦合，使递归雅可比矩阵非正规——即使算子范数超过1，所有验证过的固定点仍渐近收缩，因此真正的稳定性预算取决于谱裕度而非算子范数约束。随着携带系数ρ→1，该裕度逐渐耗尽，少数初始化无法收敛至固定点，故对角携带约束ρ(̄A)<1虽必要但不充分。六项任务上的训练实验（含受控消融）表明，线性携带并非深度记忆机制：梯度下降通过更丰富的非线性递归传递记忆，而让稳定性受限的携带保持静止——携带的作用是稳定，而非存储记忆。我们还刻画了该结论的边界：在具有轴对齐通道结构的任务上，梯度下降会动用携带。所有结果均经解析推导并由从零开始的CPU级实现验证；大规模验证仍需开展。
原文摘要 · Abstract (English)
In pre-LayerNorm looped transformers, LayerNorm inside the recurrent block acts as an implicit gain controller: by coupling the block's local Lipschitz constant inversely to the activation scale, it renders the recurrence Jacobian non-normal -- asymptotically contractive at every verified fixed point even where its operator norm exceeds 1 -- so the true stability budget is the spectral margin, not an operator-norm bound. That margin depletes as the carry $ρ\to 1$, and a minority of initializations never converge to a fixed point at all, so the diagonal carry constraint $ρ(\bar{A}) < 1$ is necessary but not sufficient for convergence of the full recurrence. Training experiments across six tasks, including a controlled ablation, reveal that the linear carry is not the depth-memory mechanism: gradient descent routes memory through the block's more expressive nonlinear recurrence and leaves the stability-constrained carry at rest -- the carry's role is stabilization, not memory. We characterize the boundary of this claim: on tasks with axis-aligned per-channel structure, gradient descent does recruit the carry. All results are derived analytically and verified in a from-scratch, CPU-scale implementation; verification at larger scale is needed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。