提出数学框架，让状态空间模型更高效压缩长期记忆。
Mathematical Formalism for Memory Compression in Selective State Space Models
- 用选择性门控动态过滤隐藏状态，实现智能记忆压缩。
- 理论证明可压缩信息量有上限，且不降低模型性能。
- 适合关注长序列建模效率的开发者和研究者。
状态空间模型（SSMs）已成为建模序列数据长程依赖的强大框架。与传统循环神经网络（RNN）和卷积神经网络（CNN）不同，SSMs 借助控制论与动力系统原理，提供结构化且稳定的序列建模方式。然而，如何在不丢失关键信息的前提下，将长程依赖压缩为紧凑的隐藏状态表示，仍是核心挑战。本文构建了针对选择性状态空间模型中记忆压缩的严格数学框架。引入选择性门控机制，根据输入相关性动态过滤与更新隐藏状态，实现高效记忆压缩。利用信息论工具（如互信息、率失真理论）形式化内存效率与信息保留之间的权衡，给出信息压缩而不影响模型性能的理论边界。推导定理证明选择性 SSM 隐藏状态的稳定性和收敛性，保障长期记忆可靠性。计算复杂度分析表明，选择性 SSM 在内存效率与处理速度上显著优于传统 RNN 模型。在时间序列预测与自然语言处理任务上的实证验证显示，该模型在使用更少内存与计算资源的情况下达到当前最优性能。
原文摘要 · Abstract (English)
State space models (SSMs) have emerged as a powerful framework for modelling long-range dependencies in sequence data. Unlike traditional recurrent neural networks (RNNs) and convolutional neural networks (CNNs), SSMs offer a structured and stable approach to sequence modelling, leveraging principles from control theory and dynamical systems. However, a key challenge in sequence modelling is compressing long-term dependencies into a compact hidden state representation without losing critical information. In this paper, we develop a rigorous mathematical framework for understanding memory compression in selective state space models. We introduce a selective gating mechanism that dynamically filters and updates the hidden state based on input relevance, allowing for efficient memory compression. We formalize the trade-off between memory efficiency and information retention using information-theoretic tools, such as mutual information and rate-distortion theory. Our analysis provides theoretical bounds on the amount of information that can be compressed without sacrificing model performance. We also derive theorems that prove the stability and convergence of the hidden state in selective SSMs, ensuring reliable long-term memory retention. Computational complexity analysis reveals that selective SSMs offer significant improvements in memory efficiency and processing speed compared to traditional RNN-based models. Through empirical validation on sequence modelling tasks such as time-series forecasting and natural language processing, we demonstrate that selective SSMs achieve state-of-the-art performance while using less memory and computational resources.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。