研究输入序列自相关对状态空间模型初始化的影响
Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models
- 基于序列自相关分析时序尺度与序列长度的关系
- 零实部特征值可缓解记忆灾难并保持初始化稳定
- 虚部特征值影响优化条件，揭示近似与估计权衡
当前状态空间模型（SSM）参数初始化主要依赖基于在线函数逼近的HiPPO框架，该框架未显式考虑输入序列时间结构对优化的影响。本文进一步探究初始化方案的作用，重点考察输入序列的自相关性：(1) 严格刻画了SSM时序尺度与序列长度之间的依赖关系；(2) 发现合理设置时序尺度下，允许状态矩阵特征值具有零实部，可在保持初始化稳定性的同时缓解记忆灾难；(3) 证明状态矩阵特征值的虚部决定优化问题的条件性，并在特定目标函数类别训练中揭示了近似与估计间的权衡关系。
原文摘要 · Abstract (English)
Current methods for initializing state space model (SSM) parameters primarily rely on the HiPPO framework \citep{gu2023how}, which is based on online function approximation with the SSM kernel basis. However, the HiPPO framework does not explicitly account for the effects of the temporal structures of input sequences on the optimization of SSMs. In this paper, we take a further step to investigate the roles of SSM initialization schemes by considering the autocorrelation of input sequences. Specifically, we: (1) rigorously characterize the dependency of the SSM timescale on sequence length based on sequence autocorrelation; (2) find that with a proper timescale, allowing a zero real part for the eigenvalues of the SSM state matrix mitigates the curse of memory while still maintaining stability at initialization; (3) show that the imaginary part of the eigenvalues of the SSM state matrix determines the conditioning of SSM optimization problems, and uncover an approximation-estimation tradeoff when training SSMs with a specific class of target functions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。