提出两种互补的注意力机制,提升模型对信息能量和位置尺度的感知能力。
Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention

- 用能量门控机制选择重要令牌,自动学习信息集中区域。
- 用莫莱特小波编码位置,自适应不同频率下的局部影响范围。
- 两者结合效果超越单独使用,适合改进小规模序列建模任务。
标准Transformer注意力计算成对标记相似性,但忽略输入的信息结构:所有标记被视为同等显著,所有位置视为同等局部。本文识别出两个标准注意力缺失的互补归纳偏置:能量显著性(哪些标记集中信息能量,通过端到端学习获得,无需显式频域分解)和尺度选择性局部性(每个频率下位置影响范围多远,通过莫莱特小波编码实现)。为此提出两个简单组件:能量门控注意力(EGA)通过单层线性投影学习键嵌入的能量估计,控制值聚合,决定关注什么;莫莱特位置编码(MoPE)以可学习的高斯窗小波替代固定正弦编码,使位置-频率联合定位适应语料库,确定在何种尺度上进行注意力操作。在TinyShakespeare上,仅使用EGA可使验证损失降低0.092(比基线+0.103);仅使用MoPE则损失上升0.032(低于基线);但二者组合达到+0.119的改进——超过各自增量之和。这一超加性效应在两次独立训练中均被观测到,是核心实证发现:显著性与局部性是互补的归纳偏置,各自填补对方无法覆盖的空白。消融实验表明,结构化的频域先验(莫莱特门控、尺度初始化头、固定正弦编码)始终弱于无约束的可学习组件,而互补的可学习组件呈现超加性交互。所有实验均为小规模(≤6M参数,字符级基准,单随机种子);未来工作最重要方向为大规模多种子验证。
原文摘要 · Abstract (English)
Standard transformer attention computes pairwise token similarity but treats all tokens as equally salient and all positions as equally local, regardless of the informational structure of the input. We identify two complementary inductive biases that standard attention lacks: energy salience (which tokens concentrate informational energy, learned end-to-end without explicit frequency decomposition) and scale-selective locality (how far positional influence extends at each frequency, implemented via Morlet wavelet encoding). We address both with two simple components. Energy-Gated Attention (EGA) gates value aggregation by a learned energy estimate of key token embeddings, computed via a single linear projection; it selects what to attend to. Morlet Positional Encoding (MoPE) replaces fixed sinusoidal encodings with learned Gaussian-windowed wavelets that adapt the joint position-frequency localization to the corpus; it specifies where attention operates at each scale. On TinyShakespeare, EGA alone achieves +0.092 validation loss improvement over standard attention (+0.103 over Phase 1-3 baseline); MoPE alone is -0.032 (below baseline as a standalone encoding); but their combination achieves +0.119 -- more than the sum of parts. This superadditivity, observed across two independent training runs, is the central empirical finding: salience and locality are complementary inductive biases, each addressing a gap the other cannot fill alone. Ablations confirm that structured spectral priors (Morlet wavelet gates, scale-initialized heads, fixed sinusoidal PE) consistently underperform their unconstrained learned counterparts, while complementary learned components interact superadditively. All experiments are at small scale (<=6M parameters, character-level benchmarks, single seed); larger-scale multi-seed validation is the most important direction for future work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。