用频谱字典替代注意力机制，实现高效可解释的语言建模。
From Attention to Atoms: Spectral Dictionary Learning for Fast, Interpretable Language Models
- 用时变傅里叶字典和词元混合系数替代传统注意力
- 在WikiText2和Penn Treebank上达到接近Transformer的困惑度
- 线性复杂度，显著降低推理延迟与内存占用
我们提出一种新型频谱生成建模范式，用于自然语言处理，联合学习全局时变傅里叶字典和每个词元的混合系数，取代Transformer架构中普遍使用的自注意力机制。通过在时域（嵌入重建）和频域（短时傅里叶变换幅值匹配）施加重构损失，并结合标准语言建模范式，同时在学习到的混合向量上拟合高斯混合模型（GMM）先验，该方法在WikiText2和Penn Treebank等标准基准上取得了具有竞争力的困惑度和生成质量。相比自注意力的二次计算复杂度，本方法实现线性复杂度，带来显著效率提升。实验表明，频谱字典模型在性能上可媲美Transformer基线，同时大幅降低推理延迟与内存开销，为可扩展语言建模提供了有力替代方案。
原文摘要 · Abstract (English)
We propose a novel spectral generative modeling framework for natural language processing that jointly learns a global time varying Fourier dictionary and per token mixing coefficients, replacing the ubiquitous self attention mechanism in transformer architectures. By enforcing reconstruction losses in both the time domain (embedding reconstruction) and the frequency domain (via Short Time Fourier Transform magnitude matching) alongside a standard language modeling objective, and fitting a Gaussian Mixture Model (GMM) prior over the learned mixing vectors, our approach achieves competitive perplexity and generation quality on standard benchmarks such as WikiText2 and Penn Treebank. In contrast to the quadratic computation complexity of self attention, our method operates with linear complexity, delivering substantial efficiency gains. We demonstrate that spectral dictionary models can achieve competitive performance compared to transformer baselines while significantly reducing inference latency and memory footprint, offering a compelling alternative for scalable language modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。