改进状态空间模型初始化，让其在有噪声数据时更稳定。
UnHiPPO: Uncertainty-aware Initialization for State Space Models
- 将数据噪声纳入理论框架，推导出考虑不确定性的初始化方法。
- 在训练和推理阶段均显著提升模型对噪声的鲁棒性。
- 无需增加计算成本，适合实际序列建模任务使用。
状态空间模型在序列建模中日益重要，许多模型依赖HiPPO框架进行动态初始化。然而，HiPPO本质上假设数据无噪声，这一假设在实践中常被违反。本文通过引入测量噪声扩展了HiPPO理论，推导出一种考虑不确定性的动态初始化方法。分析中，将HiPPO视为线性随机控制问题，原假设数据为无噪声控制信号，现重新建模为潜在系统的噪声输出，从而基于数据推断该潜在系统的后验分布，且不增加运行时间。实验表明，该初始化方法在训练和推理阶段均显著提升了状态空间模型对噪声的抵抗能力。代码已开源：https://cs.cit.tum.de/daml/unhippo。
原文摘要 · Abstract (English)
State space models are emerging as a dominant model class for sequence problems with many relying on the HiPPO framework to initialize their dynamics. However, HiPPO fundamentally assumes data to be noise-free; an assumption often violated in practice. We extend the HiPPO theory with measurement noise and derive an uncertainty-aware initialization for state space model dynamics. In our analysis, we interpret HiPPO as a linear stochastic control problem where the data enters as a noise-free control signal. We then reformulate the problem so that the data become noisy outputs of a latent system and arrive at an alternative dynamics initialization that infers the posterior of this latent system from the data without increasing runtime. Our experiments show that our initialization improves the resistance of state-space models to noise both at training and inference time. Find our implementation at https://cs.cit.tum.de/daml/unhippo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。