从S4到Mamba,解析状态空间模型如何突破序列建模的效率瓶颈。
Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mamba
- 通过结构化状态空间设计,实现线性扩展的序列处理能力。
- 在长序列任务中比Transformer更高效,推理时无需缓存增长。
- 适合需要长上下文、高吞吐的部署场景,如日志分析与超长文本生成。
结构化状态空间模型(SSMs)成为一类重要的序列模型,旨在解决循环神经网络(RNNs)的串行计算与梯度传播限制,以及Transformer自注意力机制的二次时间与内存开销问题。通过结合结构化递归与状态空间表示,SSMs实现了序列长度上的线性或近线性扩展,并在自回归解码时保持恒定大小的循环状态,无需键值缓存随上下文增长。本文系统梳理了从结构化状态空间序列模型(S4)及其对角化简化版本S4D、对角状态空间(DSS)、S5,到选择性模型Mamba和Mamba-2,再到SSM-注意力混合架构的发展脉络。分析围绕多个核心设计维度展开:输入依赖的选择性、递归与卷积视角的适用场景、对角化、推理时的缓存行为、硬件感知内核及混合设计。结构化状态空间对偶性被视为贯穿主线,解释了递归形式与注意力形式的等价性,也说明了混合架构的有效性。报告的效率与准确率结果均附带测量配置与证据等级,因为观察到的速度提升是算法、实现内核与硬件共同作用的结果,而非模型固有属性。在特定长上下文场景下,SSMs可媲美Transformer且更具效率;而在依赖精确检索与联想回忆的任务中,注意力仍具优势。论文最后讨论了现有局限、失效模式、部署考量与开放问题。
原文摘要 · Abstract (English)
Structured State Space Models (SSMs) have become a prominent class of sequence models, developed against two long-standing difficulties: the sequential computation and gradient propagation limits of Recurrent Neural Networks (RNNs), and the quadratic time and memory cost of self-attention in Transformers. By combining structured recurrence with state-space representations, SSMs attain linear or near-linear scaling in sequence length and hold a constant-size recurrent state during autoregressive decoding, so that no key-value cache grows with context. This paper is a structured review of the lineage running from the Structured State Space Sequence model (S4), through its diagonal and simplified successors S4D, Diagonal State Spaces (DSS) and S5, to the selective models Mamba and Mamba-2, and on to SSM-attention hybrids. Rather than describing models one at a time, the analysis is organized around cross-cutting design dimensions: input-dependent selectivity, the recurrent and convolutional views and when each is preferable, diagonalization, cache behavior at inference, hardware-aware kernels, and hybridization. Structured state-space duality is treated as a central thread, because it accounts for the equivalence between the recurrent and the attention-like form and for why hybrid designs work. Reported efficiency and accuracy results are presented with their measurement configuration and evidence level, because observed speedups reflect the combined effects of the algorithm, implementation kernel and hardware rather than an intrinsic property of the model alone. SSMs are competitive with Transformers and more efficient in specific long-context regimes, whereas attention retains an advantage on tasks dominated by exact retrieval and associative recall. The review closes with limitations, failure modes, deployment considerations and open problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。