改进Mamba模型在长序列任务中的表现，提升其泛化能力。
Block-Biased Mamba for Long-Range Sequence Processing
- 引入分块选择性动态与通道特异性偏置，增强模型表达力。
- 在Long-Range Arena上超越S4和S4D，在语言建模上保持原有性能。
- 适合需要长序列建模的NLP、CV任务，尤其关注稳定性与表达力的场景。
Mamba通过引入输入依赖动态扩展了早期状态空间模型（SSMs），在语言建模、计算机视觉及基础模型等领域展现出强大性能。然而，一个令人意外的弱点仍存在：尽管架构设计用于长程依赖，但Mamba在长序列任务上表现不佳。本文从可表达性、归纳偏置和训练稳定性三方面分析其局限，理论证明其相较于S4D等早期模型存在不足。为此，提出$ ext{B}_2 ext{S}_6$，即对Mamba的S6单元进行简单扩展，结合分块选择性动态与通道特异性偏置。理论证明该改进赋予模型更合适的归纳偏置，提升了表达能力和稳定性。实验表明，$ ext{B}_2 ext{S}_6$在Long-Range Arena（LRA）任务上优于S4和S4D，同时保持了Mamba在语言建模基准上的性能。
原文摘要 · Abstract (English)
Mamba extends earlier state space models (SSMs) by introducing input-dependent dynamics, and has demonstrated strong empirical performance across a range of domains, including language modeling, computer vision, and foundation models. However, a surprising weakness remains: despite being built on architectures designed for long-range dependencies, Mamba performs poorly on long-range sequential tasks. Understanding and addressing this gap is important for improving Mamba's universality and versatility. In this work, we analyze Mamba's limitations through three perspectives: expressiveness, inductive bias, and training stability. Our theoretical results show how Mamba falls short in each of these aspects compared to earlier SSMs such as S4D. To address these issues, we propose $\text{B}_2\text{S}_6$, a simple extension of Mamba's S6 unit that combines block-wise selective dynamics with a channel-specific bias. We prove that these changes equip the model with a better-suited inductive bias and improve its expressiveness and stability. Empirically, $\text{B}_2\text{S}_6$ outperforms S4 and S4D on Long-Range Arena (LRA) tasks while maintaining Mamba's performance on language modeling benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。