通过频域预处理查询键投影，显著提升字符级Transformer注意力效果。
FourierQK: Spectral Preprocessing of Query-Key Projections Improves Transformer Attention
- 在查询键投影上应用FFT频域滤波，实现全局频率混合。
- 在TinyShakespeare上最高降低1.166困惑度，性能提升79%。
- 四频段学习结果对应段落至词的多尺度结构，适合字符级建模研究者。
基于FFT的频域预处理对学习到的查询-键（Q/K）投影进行处理，在字符级语言建模任务中显著提升Transformer注意力表现。在TinyShakespeare数据集上，固定随机频域滤波器使验证困惑度降至1.031（比标准点积注意力降低0.443）；单个学习频率（段落尺度）达到0.608（降幅0.867）；四个跨段落到词尺度的学习频率组合实现0.309的困惑度（降幅1.166），相较标准注意力降低79%。该单频结果在三组随机种子下均稳定（均值0.236，标准差0.019）。四频率收敛至近似几何分布的多尺度排列（49、27、10、6词/周期），分别对应段落、子段落、短语和词尺度。性能提升源于全局频域混合而非度量扭曲：随机正交与非正交投影无改善。所有结果经洗牌验证诊断，排除位置泄露干扰。因果滤波器（高斯、墨西哥帽、莫尔莱特）在字符粒度下未优于标准注意力——本文采用双边傅里叶核，结构性地耦合前后位置。这定义了双边频域注意力与真正因果频域注意力（词粒度，配套论文MorletQK）之间的架构边界。本工作区别于FNet（Lee-Thorp et al., 2021），后者以傅里叶混合取代注意力；而本文仅对Q/K投影进行频域预处理，保留完整的注意力分数结构。
原文摘要 · Abstract (English)
FFT-based spectral preprocessing of learned query-key (Q/K) projections substantially improves transformer attention on character-level language modelling. On TinyShakespeare: a fixed random spectral filter achieves val=1.031 (Delta=+0.443); a single learned frequency at paragraph scale achieves val=0.608 (Delta=+0.867); and four learned frequencies spanning paragraph to word scale achieve val=0.309 (Delta=+1.166), a 79% reduction over standard dot-product attention. The single-frequency result is confirmed across three random seeds (mean val=0.236, std=0.019). The four frequencies converge to a near-geometric multi-scale ordering (49, 27, 10, 6 tokens/cycle) corresponding to paragraph, sub-paragraph, phrase, and word scales. The gain is specific to spectral preprocessing: random orthogonal and non-orthogonal projections of Q/K produce no measurable improvement, suggesting the benefit comes from global frequency-domain mixing rather than metric distortion. All results are verified by a shuffled-validation diagnostic against positional leakage. Causal filters (Gaussian, Mexican Hat, Morlet) do not improve over standard attention at character-level tokenisation: the bilateral FFT kernel is structurally non-causal, coupling every position to future tokens. This defines an architectural boundary between bilateral spectral attention (this paper) and genuinely causal spectral attention at word-scale tokenisation (companion paper MorletQK). This work is architecturally distinct from FNet (Lee-Thorp et al., 2021), which replaces attention with Fourier mixing of token embeddings. Here, spectral preprocessing applies only to Q/K projections while the full attention score structure is preserved.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。