arXiv:2602.01763cs.LGcs.AI2026-02被引 1

首次证明混合注意力比全注意力表达能力弱,揭示其理论局限。

A Provable Expressiveness Hierarchy in Hybrid Linear-Full Attention

  • 用递归形式统一分析线性注意力,涵盖Mamba等模型
  • 1层全注意力可解多步推理任务,混合结构需超大数量线性层
  • 为高效注意力设计提供理论依据,适合关注模型能力边界的研究者

Transformer是现代大语言模型的核心。为缓解标准全注意力的二次复杂度问题,发展出多种高效注意力机制,如线性注意力和混合注意力。但这些机制与全注意力在表达能力上的差异仍缺乏严谨的理论刻画。本文首次建立理论框架,系统分析了各类线性注意力(可表示为递归形式,包括Mamba、DeltaNet等)与全注意力的性能差异。研究发现:对于必须在前向传播中完成的序列函数复合——一种多步推理任务,(L+1)层全注意力网络即可解决;而任何将L-1层全注意力与极大数量(2^{3L^2})线性注意力层交错的混合网络均无法实现。该结果首次从理论上证明了混合注意力与全注意力之间的表达能力分离,为理解不同注意力机制的根本能力与限制提供了理论视角。

原文摘要 · Abstract (English)

Transformers serve as the foundation of most modern large language models. To mitigate the quadratic complexity of standard full attention, various efficient attention mechanisms, such as linear and hybrid attention, have been developed. A fundamental gap remains: their expressive power relative to full attention lacks a rigorous theoretical characterization. In this work, we theoretically characterize the performance differences among these attention mechanisms. Our theory applies to all linear attention variants that can be formulated as a recurrence, including Mamba, DeltaNet, etc. Specifically, we establish an expressiveness hierarchy: for the sequential function composition-a multi-step reasoning task that must occur within a model's forward pass, an ($L+1$)-layer full attention network is sufficient, whereas any hybrid network interleaving $L-1$ layers of full attention with a substantially larger number ($2^{3L^2}$) of linear attention layers cannot solve it. This result demonstrates a clear separation in expressive power between the two types of attention. Our work provides the first provable separation between hybrid attention and standard full attention, offering a theoretical perspective for understanding the fundamental capabilities and limitations of different attention mechanisms.

注意力机制理论分析Transformer表达能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。