用可学习的拉普拉斯变换替代自注意力，实现高效长文本建模。
Adaptive Two Sided Laplace Transforms: A Learnable, Interpretable, and Scalable Replacement for Self-Attention
- 提出可学习的双侧短时拉普拉斯变换，动态调整衰减率与频率响应。
- 在10万+上下文长度下表现优于或媲美现有高效模型，复杂度为O(n)。
- 兼具可解释性与扩展性，适合超长序列建模任务，如文档问答。
我们提出一种创新的、可学习的双侧短时拉普拉斯变换（STLT）机制，用于替代基于Transformer的大型语言模型中的传统自注意力。该STLT为每个拉普拉斯节点引入可训练参数，支持端到端学习衰减率、振荡频率和窗口带宽T。这种灵活性使模型能在训练中动态调整标记相关性的半衰期和频率响应。通过选择S个可学习节点并利用快速递归卷积，实现时间与内存上的有效复杂度O(n)。我们进一步引入基于FFT的高效相关矩阵计算方法及自适应节点分配机制，动态调整活跃拉普拉斯节点数量。在语言建模（WikiText-103、Project Gutenberg）、机器翻译（WMT'14 En-De）和长文档问答（NarrativeQA）上的实证结果表明，所提可学习STLT在困惑度和得分上达到或超过现有高效Transformer，自然扩展至超过10万令牌的上下文长度，仅受限于硬件资源。消融实验确认了可学习参数与自适应节点分配的重要性。该方法结合可解释性（显式衰减与频率参数）与可扩展性、鲁棒性，为无自注意力计算瓶颈的超长序列语言建模提供了新路径。
原文摘要 · Abstract (English)
We propose an innovative, learnable two-sided short-time Laplace transform (STLT) mechanism to supplant the traditional self attention in transformer-based LLMs. Our STLT introduces trainable parameters for each Laplace node, enabling end-to-end learning of decay rates , oscillatory frequencies, and window bandwidth T. This flexibility allows the model to dynamically adapt token relevance half lives and frequency responses during training. By selecting S learnable nodes and leveraging fast recursive convolution, we achieve an effective complexity of in time and memory. We further incorporate an efficient FFT-based computation of the relevance matrix and an adaptive node allocation mechanism to dynamically adjust the number of active Laplace nodes. Empirical results on language modeling (WikiText\-103, Project Gutenberg), machine translation (WMT'14 En\-De), and long document question answering (NarrativeQA) demonstrate that our learnable STLT achieves perplexities and scores on par with or better than existing efficient transformers while naturally extending to context lengths exceeding 100k tokens or more limited only by available hardware. Ablation studies confirm the importance of learnable parameters and adaptive node allocation. The proposed approach combines interpretability, through explicit decay and frequency parameters, with scalability and robustness, offering a pathway towards ultra-long-sequence language modeling without the computational bottleneck of self-attention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。