提出高阶线性注意力，实现高效长文本建模
Higher-order Linear Attention
- 用紧凑前缀统计量实现高阶交互，保持线性时间复杂度
- 二阶情况下状态大小恒定，无需存储矩阵，每标记计算线性开销
- 支持流式处理与并行训练，适合长序列建模任务
缩放点积注意力的二次复杂度是阻碍自回归语言模型扩展到长上下文的核心障碍。线性注意力与状态空间模型（SSMs）提供了可扩展的替代方案，但通常局限于一阶或基于核的近似，限制了表达能力。本文提出高阶线性注意力（HLA），一种因果、流式机制，通过紧凑前缀充分统计量实现高阶交互。在二阶情形下，HLA维持常数大小状态，以线性时间计算每个标记输出，无需显式构造任何 $n imes n$ 矩阵。我们给出闭式流式恒等式，设计了一种严格因果掩码变体，使用两个额外摘要，并提出基于结合扫描的分块并行训练方案，能精确复现串行递归的激活值。我们还概述了三阶及更高阶的扩展。这些结果使HLA成为兼具注意力式数据依赖混合与现代循环架构效率的原理性、可扩展构建块。
原文摘要 · Abstract (English)
The quadratic cost of scaled dot-product attention is a central obstacle to scaling autoregressive language models to long contexts. Linear-time attention and State Space Models (SSMs) provide scalable alternatives but are typically restricted to first-order or kernel-based approximations, which can limit expressivity. We introduce Higher-order Linear Attention (HLA), a causal, streaming mechanism that realizes higher interactions via compact prefix sufficient statistics. In the second-order case, HLA maintains a constant-size state and computes per-token outputs in linear time without materializing any $n \times n$ matrices. We give closed-form streaming identities, a strictly causal masked variant using two additional summaries, and a chunk-parallel training scheme based on associative scans that reproduces the activations of a serial recurrence exactly. We further outline extensions to third and higher orders. Collectively, these results position HLA as a principled, scalable building block that combines attention-like, data-dependent mixing with the efficiency of modern recurrent architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。