arXiv:2603.17771cs.LGcs.AI2026-03被引 1

Transformer中注意力聚集引发梯度集中,激活值自动调节训练压力。

Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformers

  • 从反向传播视角揭示注意力聚集导致梯度局部集中
  • 大激活值通过RMSNorm反向衰减梯度,缓解训练压力
  • 提出V-scale方法抑制激活值,验证梯度调节机制

注意力聚集和巨大激活值是Transformer模型中的常见且密切相关的现象。现有研究多关注前向传播,但在预归一化Transformer中,残差流范数对前向传播的影响较小,因为子层作用于归一化输入。本文从反向传播角度研究这一关系。实证与理论分析表明,在因果掩码下,注意力聚集会引发显著的梯度集中,称为梯度聚集。由于RMSNorm的雅可比矩阵大致按输入范数的倒数衰减梯度,巨大激活值可被理解为在训练过程中自适应调节局部梯度压力的机制。该解释预测:削弱由聚集引发的梯度应能抑制巨大激活。我们通过V-scale方法验证了此预测——该方法调整值路径的反向传播梯度。在V-scale模型中,注意力聚集得以保留,但巨大激活被抑制。结果表明,梯度聚集是注意力聚集的反向传播对应物,而巨大激活是受RMSNorm调节的自适应响应,用于缓解训练压力。代码已公开于https://anonymous.4open.science/r/GradientSinkCode-B309。

原文摘要 · Abstract (English)

Attention sinks and massive activations are recurring and closely related phenomena in Transformer models. Existing explanations have largely focused on the forward pass, yet in pre-norm Transformers, large residual-stream norms play only an indirect forward role because sublayers operate on normalized inputs. We study this relationship from the perspective of backpropagation. Empirically and theoretically, we show that under causal masking, attention sinks can induce pronounced gradient concentration, which we term gradient sinks. Since the RMSNorm Jacobian attenuates gradients roughly in inverse proportion to input norm, massive activations can be understood as adaptive regulators of this localized gradient pressure during training. This interpretation predicts that attenuating sink-induced gradients should weaken massive activations. We test this prediction with V-scale, a modification that adjusts backpropagated gradients on the value path. In V-scale models, attention sinks are preserved, whereas massive activations are suppressed. These results identify gradient sinks as a backward-pass counterpart of attention sinks, and massive activations as an adaptive RMSNorm-mediated response that attenuates the resulting localized training pressure. Our code is available at https://anonymous.4open.science/r/GradientSinkCode-B309.

Transformer梯度分析注意力机制训练优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。