发现状态空间模型与掩码注意力的等价关系，提升序列建模效率。
On Structured State-Space Duality
- 将状态空间模型扩展至对角矩阵形式，支持更丰富动态
- 证明其训练复杂度达到理论下限，保持线性时间优势
- 揭示仅在特定掩码下等价，不适用于标准Softmax注意力
结构化状态空间对偶性（SSD）[Dao & Gu, ICML 2024] 建立了简单结构化状态空间模型（SSM）与掩码自注意力机制之间的等价关系。具体而言，状态矩阵为标量乘单位阵的SSM等价于具有1-半可分因果掩码的掩码自注意力。因此，同一序列变换可通过线性时间 $O(T)$ 的递推或二次时间 $O(T^2)$ 的注意力实现。本文正式化并推广该对偶性：(i) 将SSD从标量-单位阵情形扩展至一般对角型SSM（对角状态矩阵）；(ii) 证明这些对角型SSM在保持训练复杂度下界的同时支持更丰富的动态；(iii) 建立了SSM与1-半可分掩码注意力等价的充要条件；(iv) 证明该对偶性无法推广至标准Softmax注意力，因存在秩爆炸问题。这些结果强化了循环式SSM与Transformer之间的桥梁，并拓展了高效且表达力强的序列模型设计空间。
原文摘要 · Abstract (English)
Structured State-Space Duality (SSD) [Dao & Gu, ICML 2024] is an equivalence between a simple Structured State-Space Model (SSM) and a masked attention mechanism. In particular, a state-space model with a scalar-times-identity state matrix is equivalent to a masked self-attention with a $1$-semiseparable causal mask. Consequently, the same sequence transformation (model) has two algorithmic realizations: as a linear-time $O(T)$ recurrence or as a quadratic-time $O(T^2)$ attention. In this note, we formalize and generalize this duality: (i) we extend SSD from the scalar-identity case to general diagonal SSMs (diagonal state matrices); (ii) we show that these diagonal SSMs match the scalar case's training complexity lower bounds while supporting richer dynamics; (iii) we establish a necessary and sufficient condition under which an SSM is equivalent to $1$-semiseparable masked attention; and (iv) we show that such duality fails to extend to standard softmax attention due to rank explosion. Together, these results tighten bridge between recurrent SSMs and Transformers, and widen the design space for expressive yet efficient sequence models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。