提出极化方法解决状态空间模型的近因偏倚与过度平滑问题。
Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing
- 通过极化状态转移矩阵的两个通道，同时缓解近因偏倚和过度平滑。
- 在长序列任务中显著提升远距离信息召回准确率，支持更深结构扩展。
- 适合关注长序列建模、状态空间模型优化的研究者与工程师。
结构化状态空间模型（SSMs）作为Transformer的替代方案，虽被认为能有效捕捉长序列依赖，但本文通过实证研究发现其存在严重的近因偏倚，导致模型难以回忆远距离信息并引发鲁棒性问题。深度扩展实验表明更深结构有助于学习长上下文，但理论分析揭示随着深度增加，模型不可避免出现过度平滑现象，即标记表示趋于不可区分。这一近因偏倚与过度平滑的根本矛盾阻碍了现有SSMs的可扩展性。受理论启发，我们提出对状态转移矩阵的两通道分别设置为0和1，实现极化，同时缓解两类问题。实验显示该方法显著提升长程标记的关联召回准确率，并使SSMs能更好地利用更深架构。所有代码已开源：https://github.com/VITA-Group/SSM-Bottleneck。
原文摘要 · Abstract (English)
Structured State Space Models (SSMs) have emerged as alternatives to transformers. While SSMs are often regarded as effective in capturing long-sequence dependencies, we rigorously demonstrate that they are inherently limited by strong recency bias. Our empirical studies also reveal that this bias impairs the models' ability to recall distant information and introduces robustness issues. Our scaling experiments then discovered that deeper structures in SSMs can facilitate the learning of long contexts. However, subsequent theoretical analysis reveals that as SSMs increase in depth, they exhibit another inevitable tendency toward over-smoothing, e.g., token representations becoming increasingly indistinguishable. This fundamental dilemma between recency and over-smoothing hinders the scalability of existing SSMs. Inspired by our theoretical findings, we propose to polarize two channels of the state transition matrices in SSMs, setting them to zero and one, respectively, simultaneously addressing recency bias and over-smoothing. Experiments demonstrate that our polarization technique consistently enhances the associative recall accuracy of long-range tokens and unlocks SSMs to benefit further from deeper architectures. All source codes are released at https://github.com/VITA-Group/SSM-Bottleneck.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。