从频域角度揭示对角状态空间模型的谱偏差，提出新初始化方法。
Uncovering the Spectral Bias in Diagonal State Space Models
- 从频率视角分析对角SSM初始化机制
- 提出S4D-DFouT初始化，显著提升性能
- 适合研究长序列建模与模型初始化的学者
当前状态空间模型（SSMs）参数初始化主要依赖基于正交多项式在线近似的HiPPO框架。近期对角变体因其在核计算上的简化而展现出相似性能与更高效率。然而，现有工作未系统研究其对角形式的作用机制。本文从频域角度深入分析对角SSM的初始化策略，揭示其内在学习偏差。基于此，提出一种在离散傅里叶域的对角初始化方法S4D-DFouT。通过合理配置极点位置，进一步实现模型扩展，在长程任务基准Long Range Arena上达到当前最优表现，并支持从零训练大规模数据集PathX-256。
原文摘要 · Abstract (English)
Current methods for initializing state space models (SSMs) parameters mainly rely on the \textit{HiPPO framework}, which is based on an online approximation of orthogonal polynomials. Recently, diagonal alternatives have shown to reach a similar level of performance while being significantly more efficient due to the simplification in the kernel computation. However, the \textit{HiPPO framework} does not explicitly study the role of its diagonal variants. In this paper, we take a further step to investigate the role of diagonal SSM initialization schemes from the frequency perspective. Our work seeks to systematically understand how to parameterize these models and uncover the learning biases inherent in such diagonal state-space models. Based on our observations, we propose a diagonal initialization on the discrete Fourier domain \textit{S4D-DFouT}. The insights in the role of pole placing in the initialization enable us to further scale them and achieve state-of-the-art results on the Long Range Arena benchmark, allowing us to train from scratch on very large datasets as PathX-256.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。