改进线性注意力模型，实现高效长序列建模且精度接近软最大注意力。
2Mamba2Furious: Linear in Complexity, Competitive in Accuracy
- 简化Mamba-2核心结构，聚焦关键设计提升表达能力。
- 新模型2Mamba在长序列上精度接近软最大注意力，内存消耗显著降低。
- 适合需要高效率长文本处理的研究者与工业应用。
线性注意力变换器因其高效性成为软最大注意力的有力替代方案。然而，线性注意力通常表达能力较弱，导致精度下降。为弥合两者之间的精度差距，本文对Mamba-2这一强线性注意力变体进行简化，识别出其最核心有效的组件。基于简化后的Mamba-2S，我们优化了A-mask并提升隐藏状态阶数，提出2Mamba方法，其精度几乎媲美软最大注意力，同时在长上下文场景下内存效率更高。此外，我们还分析了使模型超越软最大注意力精度的关键要素。所有实验代码均已开源。
原文摘要 · Abstract (English)
Linear attention transformers have become a strong alternative to softmax attention due to their efficiency. However, linear attention tends to be less expressive and results in reduced accuracy compared to softmax attention. To bridge the accuracy gap between softmax attention and linear attention, we manipulate Mamba-2, a very strong linear attention variant. We first simplify Mamba-2 down to its most fundamental and important components, evaluating which specific choices make it most accurate. From this simplified Mamba variant (Mamba-2S), we improve the A-mask and increase the order of the hidden state, resulting in a method, which we call 2Mamba, that is nearly as accurate as softmax attention, yet much more memory efficient for long context lengths. We also investigate elements to Mamba-2 that help surpass softmax attention accuracy. Code is provided for all our experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。