arXiv:2604.18580cs.LGcs.AI2026-04

将注意力嵌入循环反馈路径,实现长序列中信息的高效选择性记忆。

Sessa: Selective State Space Attention

论文配图:Sessa: Selective State Space Attention
图 1 · 摘自论文原文
  • 在循环反馈中引入多条注意力路径,替代单一注意力或递归链。
  • 理论证明其记忆衰减速率低于 $O(\ell^{-β})$,且支持不随距离衰减的选择性检索。
  • 长序列任务表现最优,短序列仍保持竞争力,适合长上下文建模场景。

现代序列建模主要依赖两类方法:Transformer 的自注意力可访问可见序列中任意元素,而结构化状态空间模型通过显式递归状态传播信息。两者在长上下文场景下面临不同局限:注意力过于分散导致单个标记影响被稀释,递归状态传播则易丢失远距离敏感性,除非主动保留信息。为此,本文提出 Sessa,一种将注意力置于循环反馈路径中的解码器。该设计使得历史标记可通过多个注意力路径影响未来状态,而非依赖单一注意力读取或单一递归链。理论上,在特定假设和匹配条件下,Sessa 可实现幂律记忆尾部 $O(\ell^{-β})$($0 < β < 1$),衰减速率快于对应 Transformer 与 Mamba 类基线。我们进一步给出实现该速率的显式构造。在同一假设下,Sessa 是唯一能实现灵活选择性检索的模型类别,包括影响不随距离衰减的特征模式。实验结果一致支持这一理论优势:在相同设置下,Sessa 在长上下文基准上表现最强,同时在短上下文语言建模任务上仍与 Transformer 及 Mamba 类模型保持竞争水平。

原文摘要 · Abstract (English)

Modern sequence modeling is dominated by two families: Transformers, whose self-attention can access arbitrary elements of the visible sequence, and structured state-space models, which propagate information through an explicit recurrent state. These mechanisms face different limitations on long contexts: when attention is diffuse, the influence of individual tokens is diluted across the effective support, while recurrent state propagation can lose long-range sensitivity unless information is actively preserved. As a result, both mechanisms face challenges in preserving and selectively retrieving information over long contexts. We propose Sessa, a decoder that places attention inside a recurrent feedback path. This creates many attention-based paths through which past tokens can influence future states, rather than relying on a single attention read or a single recurrent chain. We prove that, under explicit assumptions and matched regimes, Sessa admits power-law memory tails $O(\ell^{-β})$ for $0 < β< 1$, with slower decay than in the corresponding Transformer and Mamba-style baselines. We further give an explicit construction that achieves this power-law rate. Under the same assumptions, Sessa is the only model class among those considered that realizes flexible selective retrieval, including profiles whose influence does not decay with distance. Consistent with this theoretical advantage, across matched experiments, Sessa achieves the strongest performance on long-context benchmarks while remaining competitive with Transformer and Mamba-style baselines on short-context language modeling.

序列建模注意力机制长上下文状态空间

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。