arXiv:2509.15958cs.CLcs.LG2025-09被引 2

提出新型注意力动态模型,揭示其渐近行为规律。

Localmax dynamics for attention in transformers and its asymptotic behavior

  • 引入局部最大注意力机制,介于软最大与硬最大之间
  • 证明状态收敛至凸多面体,但结构不再由最大对齐集决定
  • 揭示静默集在系统长期行为中的关键作用,适合关注注意力理论者

我们提出一种新的离散时间注意力模型——局部最大注意力(localmax dynamics),它在经典softmax与硬最大(hardmax)之间进行插值。与硬最大类似,均匀权重由控制邻域影响的参数决定,但关键扩展在于通过一个对齐敏感度参数松弛邻域交互,实现对纯硬最大行为的可控偏离。我们证明,尽管标记状态的凸包仍收敛到凸多面体,但其结构无法再完全由最大对齐集描述,因而引入静默集(quiescent sets)来捕捉靠近顶点的标记不变行为。研究表明,即使在时变对齐敏感度参数下,这些集合仍对理解系统渐近行为起关键作用。我们进一步证明,localmax动力学不具有限时间收敛性,并给出了对消失、非零及时间变化对齐敏感度参数的分析结果,以硬最大为特例恢复其极限行为。最后,我们将基于李雅普诺夫的方法从经典意见动态中移植过来,指出其在localmax非对称设置下的局限性,并指明未来研究方向。

原文摘要 · Abstract (English)

We introduce a new discrete-time attention model, termed the localmax dynamics, which interpolates between the classic softmax dynamics and the hardmax dynamics, where only the tokens that maximize the influence toward a given token have a positive weight. As in hardmax, uniform weights are determined by a parameter controlling neighbor influence, but the key extension lies in relaxing neighborhood interactions through an alignment-sensitivity parameter, which allows controlled deviations from pure hardmax behavior. As we prove, while the convex hull of the token states still converges to a convex polytope, its structure can no longer be fully described by a maximal alignment set, prompting the introduction of quiescent sets to capture the invariant behavior of tokens near vertices. We show that these sets play a key role in understanding the asymptotic behavior of the system, even under time-varying alignment sensitivity parameters. We further show that localmax dynamics does not exhibit finite-time convergence and provide results for vanishing, nonzero, time-varying alignment-sensitivity parameters, recovering the limiting behavior of hardmax as a by-product. Finally, we adapt Lyapunov-based methods from classical opinion dynamics, highlighting their limitations in the asymmetric setting of localmax interactions and outlining directions for future research.

注意力机制动态系统理论分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。