提出零和线性注意力，让Transformer更高效且表现更强。
ZeroS: Zero-Sum Linear Attention for Efficient Transformers
- 用零和残差重加权，突破凸组合限制
- 支持正负权重，实现对比操作，长序列更稳定
- 保持线性复杂度，性能媲美甚至超越标准注意力
线性注意力方法虽将Transformer复杂度降至O(N)，但通常性能逊于标准softmax注意力。我们识别出两大根本缺陷：仅限凸组合导致信息只能相加融合，以及均匀累积权重偏差使长序列注意力衰减。为此提出零和线性注意力（ZeroS），通过移除常数零阶项1/t，并重新加权剩余的零和softmax残差，实现数学上稳定的权重分布，支持正负值，使单层注意力可执行对比操作。在保持O(N)复杂度的同时，理论上拓展了可表示函数的集合。实验表明，其在多种序列建模基准上达到或超过标准softmax注意力性能。
原文摘要 · Abstract (English)
Linear attention methods offer Transformers $O(N)$ complexity but typically underperform standard softmax attention. We identify two fundamental limitations affecting these approaches: the restriction to convex combinations that only permits additive information blending, and uniform accumulated weight bias that dilutes attention in long contexts. We propose Zero-Sum Linear Attention (ZeroS), which addresses these limitations by removing the constant zero-order term $1/t$ and reweighting the remaining zero-sum softmax residuals. This modification creates mathematically stable weights, enabling both positive and negative values and allowing a single attention layer to perform contrastive operations. While maintaining $O(N)$ complexity, ZeroS theoretically expands the set of representable functions compared to convex combinations. Empirically, it matches or exceeds standard softmax attention across various sequence modeling benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。