arXiv:2512.19428cs.LGcs.AI2025-12被引 3

不用注意力也能强推理,用几何流替代自注意力。

Attention Is Not What You Need

  • 用格拉斯曼流替代注意力矩阵,通过子空间变形传递信息。
  • 1300万参数模型在Wikitext-2上困惑度仅差Transformer的10%-15%。
  • 适合关注模型可解释性与几何结构的科研人员。

我们重新审视序列建模中的基础问题:强性能与推理是否真的需要显式自注意力?我们认为标准多头注意力本质上是张量提升:隐藏向量被映射到高维成对交互空间,学习通过梯度下降约束该提升后的张量实现。这一机制表达能力强但数学不透明,因经过多层后难以用小家族显式不变量描述模型。为此,我们提出一种无注意力架构——基于格拉斯曼流。因果格拉斯曼层(i)线性压缩标记状态,(ii)通过普鲁克坐标将局部标记对编码为格拉斯曼流形上的二维子空间,(iii)通过门控混合将这些几何特征融合回隐藏状态。信息传播依赖于多尺度局部窗口中低秩子空间的可控变形,核心计算位于有限维流形而非无结构张量空间。在Wikitext-2语言建模基准上,纯格拉斯曼模型(13至1800万参数)的验证困惑度仅比同规模Transformer低10%-15%。在SNLI自然语言推理任务中,基于格拉斯曼-普鲁克的头部在DistilBERT之上略胜于变压器头部,最佳验证与测试准确率分别为0.8550和0.8538,优于对比组的0.8545和0.8511。我们分析了格拉斯曼混合的复杂度,证明其在固定秩下序列长度呈线性增长,并主张此类流形设计为神经推理的几何与不变性解释提供了更结构化的路径。

原文摘要 · Abstract (English)

We revisit a basic question in sequence modeling: is explicit self-attention actually necessary for strong performance and reasoning? We argue that standard multi-head attention is best seen as a form of tensor lifting: hidden vectors are mapped into a high-dimensional space of pairwise interactions, and learning proceeds by constraining this lifted tensor through gradient descent. This mechanism is extremely expressive but mathematically opaque, because after many layers it becomes very hard to describe the model with a small family of explicit invariants. To explore an alternative, we propose an attention-free architecture based on Grassmann flows. Instead of forming an L by L attention matrix, our Causal Grassmann layer (i) linearly reduces token states, (ii) encodes local token pairs as two-dimensional subspaces on a Grassmann manifold via Plucker coordinates, and (iii) fuses these geometric features back into the hidden states through gated mixing. Information therefore propagates by controlled deformations of low-rank subspaces over multi-scale local windows, so the core computation lives on a finite-dimensional manifold rather than in an unstructured tensor space. On the Wikitext-2 language modeling benchmark, purely Grassmann-based models with 13 to 18 million parameters achieve validation perplexities within about 10 to 15 percent of size-matched Transformers. On the SNLI natural language inference task, a Grassmann-Plucker head on top of DistilBERT slightly outperforms a Transformer head, with best validation and test accuracies of 0.8550 and 0.8538 compared to 0.8545 and 0.8511. We analyze the complexity of Grassmann mixing, show linear scaling in sequence length for fixed rank, and argue that such manifold-based designs offer a more structured route toward geometric and invariant-based interpretations of neural reasoning.

注意力机制几何深度学习序列建模可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。