arXiv:2602.01744cs.LGcs.AI2026-02被引 1

让线性注意力重新获得全局竞争能力,提升长文本理解精度。

Softmax Linear Attention: Reclaiming Global Competition

  • 将softmax从令牌层升至头层,用多头作为语义槽实现动态竞争选择。
  • 在长文本任务中显著提升检索鲁棒性,优于现有线性注意力基线。
  • 适合需要高效又精准处理长上下文的场景,如文档理解、问答系统。

尽管线性注意力将标准Transformer的二次复杂度降至线性时间,但因移除softmax归一化而表达能力下降,丧失了关键的‘全局竞争’机制,难以在长上下文噪声中聚焦关键信息。本文提出软最大线性注意力(SLA),通过将softmax操作从令牌层面提升至头层面,使注意力头作为粗粒度语义槽,引入竞争门控机制动态选择最相关子空间,重新引入‘赢家通吃’的动态特性,实现精确检索与稳健的长上下文理解。不同于以往优化局部核函数的方法,SLA从更高层次的多头聚合结构出发,具有更广视角。大量实验表明,SLA在语言建模与长上下文基准上持续超越当前主流线性基线模型(RetNet、GLA、GDN),尤其在高噪声检索任务中表现显著提升,验证其在保持线性复杂度前提下恢复精准聚焦的能力。

原文摘要 · Abstract (English)

While linear attention reduces the quadratic complexity of standard Transformers to linear time, it often lags behind in expressivity due to the removal of softmax normalization. This omission eliminates \emph{global competition}, a critical mechanism that enables models to sharply focus on relevant information amidst long-context noise. In this work, we propose \textbf{Softmax Linear Attention (SLA)}, a framework designed to restore this competitive selection without sacrificing efficiency. By lifting the softmax operation from the token level to the head level, SLA leverages attention heads as coarse semantic slots, applying a competitive gating mechanism to dynamically select the most relevant subspaces. This reintroduces the ``winner-take-all'' dynamics essential for precise retrieval and robust long-context understanding. Distinct from prior methods that focus on refining local kernel functions, SLA adopts a broader perspective by exploiting the higher-level multi-head aggregation structure. Extensive experiments demonstrate that SLA consistently enhances state-of-the-art linear baselines (RetNet, GLA, GDN) across language modeling and long-context benchmarks, particularly in challenging retrieval scenarios where it significantly boosts robustness against noise, validating its capability to restore precise focus while maintaining linear complexity.

线性注意力长上下文多头竞争

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。