arXiv:2508.03158cs.LGcs.IT2025-08被引 2

提出新原则让状态空间模型更智能地筛选信息,提升预测准确性和鲁棒性。

Rethinking Selectivity in State Space Models: A Minimal Predictive Sufficiency Approach

  • 基于信息论设计最小预测充分性原则,指导模型筛选关键历史信息。
  • 在长序列预测和噪声场景下表现优于现有模型,准确率显著提升。
  • 可作为通用正则化框架,适用于多种主流序列模型,适用性强。

状态空间模型(SSMs)特别是近年的可选变体(如Mamba)已成为序列建模的领先架构,挑战了Transformer的主导地位。然而,这些先进模型的成功很大程度上依赖于启发式设计的选择机制,缺乏严格的原理推导。这一理论缺口引发了对其最优性和对抗虚假相关性的鲁棒性的质疑。为此,我们提出预测充分性原则,即理想隐藏状态应是过去信息对预测未来最简充分统计量。基于此,我们构建了最小预测充分性状态空间模型(MPS-SSM),其选择机制由该原则导出的目标函数驱动。该方法鼓励模型在不损失预测能力的前提下最大化压缩历史信息,从而学会忽略非因果噪声与虚假模式。大量实验表明,MPS-SSM不仅在多个基准数据集上达到顶尖性能,尤其在长期预测与噪声环境下显著超越现有模型,且展现出更强鲁棒性。此外,我们证明该原则可扩展为通用正则化框架,用于增强其他主流架构,凸显其广泛潜力。

原文摘要 · Abstract (English)

State Space Models (SSMs), particularly recent selective variants like Mamba, have emerged as a leading architecture for sequence modeling, challenging the dominance of Transformers. However, the success of these state-of-the-art models largely relies on heuristically designed selective mechanisms, which lack a rigorous first-principle derivation. This theoretical gap raises questions about their optimality and robustness against spurious correlations. To address this, we introduce the Principle of Predictive Sufficiency, a novel information-theoretic criterion stipulating that an ideal hidden state should be a minimal sufficient statistic of the past for predicting the future. Based on this principle, we propose the Minimal Predictive Sufficiency State Space Model (MPS-SSM), a new framework where the selective mechanism is guided by optimizing an objective function derived from our principle. This approach encourages the model to maximally compress historical information without losing predictive power, thereby learning to ignore non-causal noise and spurious patterns. Extensive experiments on a wide range of benchmark datasets demonstrate that MPS-SSM not only achieves state-of-the-art performance, significantly outperforming existing models in long-term forecasting and noisy scenarios, but also exhibits superior robustness. Furthermore, we show that the MPS principle can be extended as a general regularization framework to enhance other popular architectures, highlighting its broad potential.

状态空间模型序列建模信息论鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。