提出新方法分析无归一化注意力模型的信号传播，揭示其训练稳定性机制。
Subcritical Signal Propagation at Initialization in Normalization-Free Transformers
- 用平均部分雅可比范数量化梯度放大，推导出多层注意力模型的递推关系
- 发现替换层归一化后信号增长从幂律转为伸展指数，表明系统进入亚临界状态
- 解释动态激活函数模型对初始化敏感的原因，适合关注训练稳定性的研究者
我们通过平均部分雅可比范数（APJN）研究变换器在初始化时的信号传播，该指标衡量层间梯度放大程度。通过推导激活统计量和APJN在各层间的递推关系，将APJN分析扩展至具有双向注意力和排列对称输入的变换器。理论预测了注意力对深层情况下APJN渐近行为的影响，并与深度视觉变换器中的实测结果一致。残差网络中的临界性图像同样适用于变换器：预层归一化架构呈现幂律增长，而用逐元素tanh类非线性替代层归一化的架构则呈现伸展指数增长，表明后者处于亚临界状态。该理论成功解释了动态tanh（DyT）和动态erf（Derf）变换器为何对初始化和优化选择更敏感，且需精细调参以实现稳定训练。
原文摘要 · Abstract (English)
We study signal propagation at initialization in transformers through the averaged partial Jacobian norm (APJN), a measure of gradient amplification across layers. We extend APJN analysis to transformers with bidirectional attention and permutation-symmetric input token configurations by deriving recurrence relations for activation statistics and APJNs across layers. Our theory predicts how attention modifies the asymptotic behavior of the APJN at large depth and matches APJNs measured in deep vision transformers. The criticality picture known from residual networks carries over to transformers: the pre-LayerNorm architecture exhibits power-law APJN growth, whereas transformers with LayerNorm replaced by elementwise $\tanh$-like nonlinearities have stretched-exponential APJN growth, indicating that the latter are subcritical. Applied to Dynamic Tanh (DyT) and Dynamic erf (Derf) transformers, the theory explains why these architectures can be more sensitive to initialization and optimization choices and require careful tuning for stable training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。