将状态空间模型的重要性信号融入注意力机制，实现高效精准的文本建模。
Forget Attention: Importance-Aware Attention Is All You Need
- 在注意力分数中直接加入SSM计算的重要度信号，实现端到端融合。
- 在152M参数下，LAMBADA准确率达17.3%，比Transformer高3.4个百分点。
- 无需循环状态或自定义核，兼容标准SDPA，适合大规模部署。
将注意力机制的全局检索能力与状态空间模型（SSMs）的序列重要性信号相结合，是混合语言建模的开放挑战。变压器模型能关注全局却无法优先处理关键信息；而SSMs虽知何者重要，却无法回溯。现有混合模型——Jamba（块级）和Hymba（头级）——将两者分置于独立模块，导致注意力计算过程中无法相互影响。本文提出SISA（SSM-Informed Softmax Attention），通过在注意力分数中直接引入由SSM生成的重要性项，实现完整运算仅需一次标准SDPA调用，无需递归状态，也无需定制内核。在152M/5B tokens训练下，SISA在LAMBADA-greedy任务上达到17.3%准确率（超越Transformer的13.9%和Mamba-3的15.5%），并在第1000步即达成100%的NIAH性能，较变压器快7倍；在369M参数规模下，尽管Mamba-3领先于LAMBADA，SISA仍保持完美NIAH表现并兼容标准SDPA执行。SISA因此确立了第三种SSM-注意力混合范式——分数级融合，突破了此前块级与头级主导的设计格局。
原文摘要 · Abstract (English)
Combining attention's global retrieval with the sequential importance signal of state space models (SSMs) is the open challenge of hybrid language modeling. Transformers see everywhere but cannot prioritize; SSMs know what matters but cannot revisit. Existing hybrids -- Jamba (block level) and Hymba (head level) -- place the two in separate compartments, so neither informs the other during the attention computation itself. We propose SISA (SSM-Informed Softmax Attention), which adds an SSM-derived importance term directly inside the attention score and realizes the full operation as a single SDPA call on augmented query/key vectors -- no recurrent state, no custom kernel. At 152M / 5B tokens, SISA reaches LAMBADA-greedy 17.3% (vs. Transformer 13.9 and Mamba-3 15.5) and attains NIAH 100% from step 1K, 7x faster than Transformer's retrieval convergence; at 369M, Mamba-3 leads LAMBADA while SISA preserves perfect NIAH and stock-SDPA execution. SISA thus defines a third design axis for SSM-attention hybrids -- score-level fusion -- beyond the block-level and head-level paradigms that have dominated the field.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。