通过尺度选择性分解，揭示Transformer各层注意力的层次结构。
Multiscale POD of Transformer Attention Fields: Scale-Selective Analysis via Morlet Scalogram
- 用小波变换识别注意力时序尺度，再对每尺度做主成分分析。
- 早期层关注精细尺度，后期层转向粗粒度模式，与层深相关。
- 无需修改模型或标注，仅靠统计就能量化各层注意力复杂度。
我们提出一种针对Transformer注意力场的尺度选择性奇异正交分解（POD），灵感来自其在湍流集合中提取能量主导模态的应用。通过莫雷特连续小波变换，识别文档集合中注意力滞后的主导时间尺度；随后，对每个尺度下的注意力场集合进行POD，提取该尺度的能量主导模态。结果表明，各层表现出分层的尺度组织：早期层侧重精细尺度，后期层趋向粗粒度尺度。我们基于POD特征值衰减率定义了谱集中指数，并实证显示其能有效区分不同层的注意力场复杂度。根据经典POD最优性定理，所提取模态在集合上最小化平均L2重建误差（定理1），为每层提供数据驱动的有效秩。该方法无需架构修改或语言标注，主导注意力模式完全由集合统计产生。湍流类比仅在于集合协方差与模态分析结构，而非物理机制。
原文摘要 · Abstract (English)
We introduce scale-selective Proper Orthogonal Decomposition (POD) for transformer attention fields, inspired by the use of POD for extracting energetically dominant modes from turbulent flow ensembles. The Morlet continuous wavelet transform identifies dominant temporal scales in the attention lag structure across a document ensemble; POD then extracts the energetically dominant modes at each scale from the ensemble of attention fields. The resulting modes reveal layer-dependent scale organisation, with early layers emphasising fine scales and later layers shifting toward coarser scales. We define a spectral concentration index from the POD eigenvalue decay rate and show empirically that it differentiates layers by their attention field complexity. By the classical POD optimality theorem, the extracted modes minimise the average L2 reconstruction error over the ensemble (Theorem 1), giving a data-driven effective rank for each layer. The method requires no architectural modification and no linguistic annotations: dominant attention patterns emerge from ensemble statistics alone. The turbulence analogy is structural rather than physical: we borrow ensemble covariance and modal analysis, not fluid dynamics itself.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。