arXiv:2602.21340cs.LG2026-02被引 1

让状态空间模型的记忆机制可解释,用多项式显式建模历史。

HiPPO Zoo: Explicit Memory Mechanisms for Interpretable State Space Models

  • 基于正交多项式显式构建记忆,直接可解释。
  • 支持自适应记忆分配和关联记忆,保持高效在线更新。
  • 适合需要透明决策过程的长序列任务,如医疗、金融分析。

在序列数据训练系统中,如何以压缩、高效且信息丰富的方式表示过去是核心挑战。HiPPO框架通过结构化线性常微分方程将信号投影到正交多项式(OP)基上,提供了一种原理严谨的序列压缩方法。后续工作将这些动态嵌入状态空间模型(SSMs),其中HiPPO结构作为初始化。尽管非线性改进版如Mamba在处理长程依赖任务上达到前沿水平,但其记忆机制仍隐含难解。本文重新审视HiPPO框架,旨在使其记忆机制显式可解释。我们展示了如何将历史的多项式表示扩展至支持现代SSMs的能力,如自适应记忆分配和关联记忆,同时保留在OP基下的直接可解释性。提出一个统一框架,包含五种扩展,统称“HiPPO zoo”。每种扩展通过显式、可解释的方式修改HiPPO框架,实现特定建模能力。所提模型可在线调整记忆并支持流式训练,更新高效。通过一系列合成序列建模任务,验证了这些扩展的行为与优势,证明通常归因于现代SSMs的能力,可通过显式、可解释的多项式记忆结构实现。

原文摘要 · Abstract (English)

Representing the past in a compressed, efficient, and informative manner is a central problem for systems trained on sequential data. The HiPPO framework, originally proposed by Gu & Dao et al., provides a principled approach to sequential compression by projecting signals onto orthogonal polynomial (OP) bases via structured linear ordinary differential equations. Subsequent works have embedded these dynamics in state space models (SSMs), where HiPPO structure serves as an initialization. Nonlinear successors of these SSM methods such as Mamba are state-of-the-art for many tasks with long-range dependencies, but the mechanisms by which they represent and prioritize history remain largely implicit. In this work, we revisit the HiPPO framework with the goal of making these mechanisms explicit. We show how polynomial representations of history can be extended to support capabilities of modern SSMs such as adaptive memory allocation and associative memory, while retaining direct interpretability in the OP basis. We introduce a unified framework comprising five such extensions, which we collectively refer to as a "HiPPO zoo." Each extension exposes a specific modeling capability through an explicit, interpretable modification of the HiPPO framework. The resulting models adapt their memory online and train in streaming settings with efficient updates. We illustrate the behaviors and modeling advantages of these extensions through a range of synthetic sequence modeling tasks, demonstrating that capabilities typically associated with modern SSMs can be realized through explicit, interpretable polynomial memory structures.

状态空间模型可解释性记忆机制序列建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。