arXiv:2605.21842cs.LGcs.CL2026-05被引 4

让注意力聚焦信息密集的关键词,提升模型表现

Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention

论文配图:Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention
图 1 · 摘自论文原文
  • 用频谱能量动态调节注意力权重,识别关键信息位置
  • 在TinyShakespeare上降低验证损失0.103,仅增1.2万参数
  • 发现英语中约36%的词携带高能量,适配语言结构分析

标准Transformer注意力对所有词元平等对待,忽视其内在信息含量。在湍流流体动力学中,相干结构虽处于背景混沌中,却承载绝大部分能量并主导输运过程。我们提出:词元在注意力中也扮演类似角色——信息密集位置(如句法核心、语义边界)集中频谱能量,应获得更高关注。为此提出能量门控注意力(EGA):通过单个可学习线性投影计算键词元嵌入的频谱能量,以门控值聚合。在TinyShakespeare上,EGA使验证损失降低0.103,仅增加12,480参数(<0.26%开销),计算成本无显著增加。该结果在Penn Treebank上同样有效(+0.101)。三类小波基的系统消融表明,固定结构基非最优——最佳能量方向是数据自适应且非正弦的;而学习的小波包是一个有前景的方向。学习到的能量阈值收敛至τ≈0.35,对应英文文本中约36%的词具有高于平均的频谱能量,这一比例稳定,与英语内容词占比一致。

原文摘要 · Abstract (English)

Standard transformer attention computes pairwise similarity between queries and keys, treating all tokens as equally salient regardless of their intrinsic informational content. In turbulent fluid dynamics, coherent structures -- the energetically dominant, spatially organized patterns that persist amid background chaos -- carry a disproportionate fraction of total energy and govern all transport. We propose that tokens play an analogous role in transformer attention: informationally dense positions (morphological boundaries, syntactic heads, discourse markers) concentrate spectral energy and should attract proportionally more attention than background tokens (function words, repeated patterns, low-information filler). We propose Energy-Gated Attention (EGA): a simple modification that gates value aggregation by the spectral energy of key token embeddings, computed by a single learned linear projection that discovers the dominant spectral mode of the embedding field. On TinyShakespeare, EGA achieves +0.103 validation loss improvement with only 12,480 additional parameters (<0.26% overhead) and no measurable computational cost. The result is consistent on Penn Treebank (+0.101), demonstrating dataset independence. A systematic ablation across three wavelet families (fixed Morlet, Daubechies db2/db4, and a parametric Morlet) establishes that fixed structured bases are suboptimal -- the optimal energy direction is data-adaptive and non-sinusoidal -- while identifying learned wavelet packets as a promising open direction. The learned energy threshold converges to tau ~= 0.35 independently of initialization, corresponding to the fraction (~36%) of tokens carrying above-average spectral energy in English text, a stable linguistic property consistent with the fraction of content words in running English text.

注意力机制频谱分析语言结构模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。