提出精确线性注意力,实现高效无误差的长序列建模。
Exact Linear Attention

- 通过核函数精确分解,实现线性复杂度且无近似误差。
- 解码速度提升6倍,KV缓存占用减少75%,训练性能相当或更优。
- 适合长序列建模与视觉任务,尤其适用于资源受限场景。
本文提出精确线性注意力(ELA),通过利用核函数的精确分解特性,实现Transformer注意力的线性计算复杂度,彻底消除近似误差。针对先前线性注意力存在的梯度爆炸和令牌注意力稀释问题,引入核函数约束以确保非负性、可区分性和几何可解释性,提出了哈达玛指数核、求和平方欧氏距离核、差值平方欧氏距离核等,分别适配不同注意力行为。此外,提出三项工程创新:(1) 用超链接结构替代残差连接,缓解梯度退化;(2) 基于双向线性注意力的记忆叶模块,捕捉层间“变换流”,实现定性记忆与隐式强化学习;(3) 基于路由得分的专家混合(MoE)偏置机制,提升可解释性与语义对齐。实验表明,ELA相比全注意力实现最高6倍解码加速、75% KV缓存内存降低,训练表现相当或更优;所提记忆模块加速收敛并提升泛化能力。进一步将线性注意力原则拓展至视觉模型,构建YOLO-LAT,实现最高4.3倍GPU推理加速与7.9倍参数压缩,检测精度保持竞争力。结果表明,精确线性注意力在超长序列与高效视觉任务中具有广泛适用性。
原文摘要 · Abstract (English)
This paper introduces Exact Linear Attention (ELA), a mechanism that achieves linear computational complexity for Transformer attention by exploiting the exact decomposition property of kernel functions, thereby eliminating approximation error. We identify and address two key limitations of prior linear attention -- gradient explosion and token attention dilution -- by imposing kernel constraints that ensure non-negativity, discriminability, and geometric interpretability. Several kernel functions are proposed, including the Hadamard Exp Kernel, Summation Squared Euclidean Distance Kernel, and Subtraction Squared Euclidean Distance Kernel, each tailored for specific attention behaviors. Beyond the core attention formulation, the paper presents three engineering innovations: (1) a Hyper-Link structure that replaces traditional residual connections to mitigate gradient degradation; (2) a Memory Lobe module based on bidirectional linear attention, which captures "transformation flow" across layers to implement qualitative memory and an implicit reinforcement learning paradigm; and (3) a routing-score-based bias mechanism for Mixture-of-Experts (MoE) to improve interpretability and semantic alignment. Experimental results demonstrate that ELA achieves up to 6x faster decoding speed and 75% reduction in KV cache memory usage compared to full attention, while maintaining comparable or superior training performance. The proposed memory module accelerates convergence and enhances generalization. Furthermore, we extend the linear attention principle to vision models, yielding YOLO-LAT, which attains up to 4.3x GPU inference speedup and 7.9x parameter reduction with competitive detection accuracy. These results underline the broad applicability of exact linear attention for scaling Transformer models to ultra-long sequences and efficient visual tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。