证明了状态跟踪需同时具备输入依赖和负特征值的SSM结构
Parity Requires Unified Input Dependence and Negative Eigenvalues in SSMs
- 提出统一输入依赖与负特征值的SSM设计要求
- 验证组合S4D与Mamba层仍无法解决奇偶性任务
- 为高效SSM模型提供理论设计指导，适合序列建模研究者
近期研究表明，S4D、Mamba和DeltaNet等LRNN模型因过渡矩阵时不变或特征值范围受限，缺乏状态追踪能力。为此，已有研究提出输入依赖的过渡矩阵（如复数或非三角矩阵）以提升SSM性能。尽管现有定理表明，无论深度如何，仅输入独立或非负特征值的SSM均无法解决简单状态追踪任务（如奇偶性），但未探讨多层SSM中两类结构组合是否可行。本文针对具有对角过渡矩阵的高效SSM进行研究，发现此类组合仍无法解决奇偶性问题。这表明，递归层必须同时具备输入依赖性和负特征值。实验通过分析结合S4D与Mamba层的SSM模型验证了该结论。
原文摘要 · Abstract (English)
Recent work has shown that LRNN models such as S4D, Mamba, and DeltaNet lack state-tracking capability due to either time-invariant transition matrices or restricted eigenvalue ranges. To address this, input-dependent transition matrices, particularly those that are complex or non-triangular, have been proposed to enhance SSM performance on such tasks. While existing theorems demonstrate that both input-independent and non-negative SSMs are incapable of solving simple state-tracking tasks, such as parity, regardless of depth, they do not explore whether combining these two types in a multilayer SSM could help. We investigate this question for efficient SSMs with diagonal transition matrices and show that such combinations still fail to solve parity. This implies that a recurrence layer must both be input-dependent and include negative eigenvalues. Our experiments support this conclusion by analyzing an SSM model that combines S4D and Mamba layers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。