用小波理论改进位置编码，让模型自适应学习词语影响范围。
Beyond Sinusoids: A Morlet Wavelet Framework for Transformer Positional Encoding
- 每维嵌入自主学习频率与局部性带宽，取代固定正弦编码。
- 在TinyShakespeare上比标准注意力提升0.119，且优于单独组件。
- 参数收敛至小波容许性边界，揭示字符级语言信号的潜在规律。
Transformer的标准位置编码（正弦与旋转编码）将每个位置视为同等局部：它们记录了词元的位置，但未体现其影响范围。本文提出，同时最小化位置与频率不确定性的Morlet小波是位置编码的自然基础，并引入Morlet位置编码（MoPE）：每个嵌入维度从数据中学习自身的频率与局部性带宽。主要理论结果为统一性：当局部性关闭（sigma_i → ∞）时，正弦编码与RoPE相关核均成为MoPE的极限情况。MoPE的相位精确恢复RoPE旋转角；振幅则添加了一个可学习的高斯局部核，这是标准编码所缺失的。实验表明，结合能量门控注意力的MoPE在TinyShakespeare上比标准注意力提升0.119，且优于任一组件单独使用。对学习参数的分析显示，全部128个频率-带宽对均收敛至小波容许性边界——这一经验观察与能量门控的配套结果一致，暗示字符级语言信号存在可复现的特性，值得进一步研究。
原文摘要 · Abstract (English)
Standard positional encodings for transformers - sinusoidal and rotary (RoPE) - treat every position as equally local: they encode where a token is, but not how far its positional influence should extend. We propose that the Morlet wavelet, which simultaneously minimises uncertainty in position and frequency, is the natural basis for positional encoding, and introduce Morlet Positional Encoding (MoPE): each embedding dimension learns its own frequency and locality bandwidth from data. The main theoretical result is a unification: sinusoidal PE and the RoPE correlation kernel both emerge as limiting cases of MoPE when locality is switched off (sigma_i -> infinity). The phase of MoPE recovers the RoPE rotation angle exactly; the amplitude adds a learned Gaussian locality kernel that standard encodings lack. Empirically, MoPE combined with Energy-Gated Attention achieves +0.119 improvement over standard attention on TinyShakespeare, outperforming either component alone. Analysis of the learned parameters reveals that all 128 frequency-bandwidth pairs converge to the wavelet admissibility boundary - an empirical observation consistent with a companion result on energy gating, suggesting a reproducible property of character-level language signals that warrants further investigation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。