arXiv:2602.08216cs.LGcond-mat.stat-mech2026-02被引 2

用热力学视角揭示注意力机制的内在规律,发现其与统计物理系统的相似性。

Thermodynamic Isomorphism of Transformers: A Lagrangian Approach to Attention Dynamics

  • 构建基于费雪信息度量的拉格朗日框架,将注意力看作最小化自由能的稳定解。
  • 在模加法任务中观察到注意力能量方差峰值,该现象先于泛化出现。
  • 为理解模型训练动态和位置编码提供统一的统计力学解释,适合关注理论机制的研究者。

我们提出一种有效场论框架,从热力学角度分析Transformer注意力机制。通过在配备费雪度量的信息流形上构建拉格朗日量,在香农-玻尔兹曼熵框架下,证明缩放点积注意力可视为最小化赫尔姆霍兹自由能泛函的驻定解,建立起其与经典系综统计的正式对应关系。进一步将该映射扩展至宏观可观测量,定义了与注意力能量景观涨落相关联的有效比热。在模加法任务(p = 19–113)的受控实验中,观察到能量方差存在稳健峰值,且始终先于泛化出现。尽管在有限深度下未检测到渐近幂律发散,但能量方差的可重复增强表明伴随表征重组出现类临界交叉行为。该框架为注意力缩放、训练动态及位置编码提供了统一的统计力学视角,将其解释为有效热力学系统的涌现特性,而非孤立启发式。当前结果表明有限尺寸交叉行为而非严格相变,但仍激励通过涨落可观测量探索深层架构的标度极限。

原文摘要 · Abstract (English)

We propose an effective field-theoretic framework for analyzing Transformer attention through a thermodynamic lens. By constructing a Lagrangian on the information manifold equipped with the Fisher metric, we show that, within the Shannon--Boltzmann entropy framework, the Softmax function arises as a stationary solution minimizing a Helmholtz free energy functional. This establishes a formal correspondence between scaled dot-product attention and canonical ensemble statistics. Extending this mapping to macroscopic observables, we define an effective specific heat associated with fluctuations of the attention energy landscape. In controlled experiments on the modular addition task ($p = 19$--$113$), we observe a robust peak in this fluctuation measure that consistently precedes the onset of generalization. While no asymptotic power-law divergence is detected in this finite-depth regime, the reproducible enhancement of energy variance suggests a critical-like crossover accompanying representational reorganization. Our framework provides a unified statistical-mechanical perspective on attention scaling, training dynamics, and positional encoding, interpreting the phenomena as emergent properties of an effective thermodynamic system rather than isolated heuristics. Although the present results indicate finite-size crossover behavior rather than a strict phase transition, they motivate further investigation into scaling limits of deep architectures through fluctuation-based observables.

注意力机制统计物理模型动力学

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。