揭示注意力分布对Transformer局部Lipschitz常数的影响，提升模型训练稳定性。
Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers
- 基于softmax雅可比奇异值的紧致上界，分析注意力分布对模型敏感度的影响。
- 提出新上界，比已有方法更紧致，且首次显式关联注意力概率分布与模型稳定性。
- 设计轻量正则化器JaSMin，直接控制每层局部Lipschitz常数，适合追求稳定训练的开发者。
本文提出了点积自注意力模块局部Lipschitz常数的新上界，揭示了其对注意力图分布的依赖关系。该上界不仅比现有方法更紧致，更是首次阐明注意力概率分布如何影响自注意力块的局部Lipschitz常数。理论基础源于对softmax函数雅可比奇异值的精细化闭式上界分析。基于此，我们提出JaSMin（Jacobian Softmax norm Minimization），一种轻量级正则化器，可直接调控每一层的局部Lipschitz常数，从而影响整个模型的性质。此外，还讨论了注意力分布特性对梯度动态及变压器训练稳定性的作用。
原文摘要 · Abstract (English)
We introduce a novel upper bound on the local Lipschitz constant of the dot-product self-attention block showing its dependence on the attention map distributions. The proposed bound is not only tighter than the prior art, but for the first time, reveals how the distribution of attention probabilities shapes the local Lipschitz constant of the self-attention block. The theoretical basis of the proposed upper bound lies in the refined closed-form upper bounds on singular values of the Jacobian of softmax function. Leveraging these theoretical insights, we introduce JaSMin (Jacobian Softmax norm Minimization), a lightweight regularizer that directly controls the local Lipschitz constant of each block and, consequently, the entire model. Additionally, we discuss how the nature of the attention map distribution contributes to the gradient dynamics and, consequently, transformer training stability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。