arXiv:2509.04226cs.LG2025-09被引 1

LISM让状态空间模型记住更长时间,解决长期依赖难题。

LISM: Long-range Integrative State space Models via Input-Latent State Interactions

  • 允许部分不稳定的递归更新,增强记忆保持能力
  • 在选择性复制和生物预测任务中表现最优,语言任务也更稳定
  • 适合需要长程记忆和高重复性的序列建模场景

状态空间模型(SSMs)虽具线性时间复杂度,但存在固有的“记忆诅咒”——如Mamba等模型的内存会随稳定递归而指数衰减。本文提出长程整合状态空间模型(LISM),通过允许递归更新偶尔不稳定,实现更长的记忆保留,并引入动态输入-状态交互机制,识别需长期记忆的输入。理论推导了整体稳定性概率边界与概率稳定性条件,指导参数化与训练。在合成选择性复制任务、真实语言任务WikiText-103和LRA Retrieval,以及生物预测任务上评估,LISM在选择性复制、WikiText-103和生物预测任务上优于其他基于SSM的模型,在LRA-Retrieval上达到领先水平。此外,其性能波动显著降低:在两个真实语言任务中,相对性能方差平均减少33.3%,在WikiText-103上标准差为±0.30%,优于Mamba-2的±0.46%。代码已开源。

原文摘要 · Abstract (English)

State space models (SSMs) are an emerging paradigm that achieves linear-time scaling, however, they intrinsically suffer from "curse of memory". The memory of SSMs, including Mamba, decays exponentially as long as the recursive update is stable. In this work, we developed Long-range Integrative State space Models (LISM) to mitigate the curse of memory in SSMs. LISM allows the recursive update to occasionally be unstable, which enables longer memory retention, and incorporates a dynamic input-state interaction mechanism to identify inputs that should be memorized for longer periods. We theoretically establish probability bounds for the overall stability of LISM and derive a probabilistic stability condition to guide parameterization and training. We evaluated LISM's performance on multiple tasks, a synthetic Selective Copy task, two real language tasks WikiText-103 and LRA Retrieval, and an additional biological prediction task. LISM achieved the best performance on the Selective Copying task, WikiText-103, and the biological prediction task compared to SSM-based models and achieved comparable performance on the LRA-Retrieval task with the state-of-the-art. Beyond average performance, LISM also shows strong repeatability across runs. In the two real language tasks, LISM reduced the relative performance variance (coefficient of variation) by an average of 33.3%, achieving a standard deviation of $\pm$0.30% on WikiText-103 compared to $\pm$0.46% for Mamba-2. The implementation of LISM is available at https://github.com/CM-Compbio-Group/LISM_dev.

状态空间模型长程记忆稳定性分析序列建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。