用小波框架重构状态矩阵，让序列模型更长程建模更稳定。
W4S4: WaLRUS Meets S4 for Long-Range Sequence Modeling
- 基于小波框架设计新状态矩阵，实现稳定对角化与快速计算。
- 在长序列任务中性能超越基于HiPPO的模型，深度架构中仍保持优势。
- 适合需要高精度长程依赖建模的研究者与工业应用。
状态空间模型（SSMs）已成为高效处理长程依赖的有力工具，通过线性递推与卷积计算实现。然而其性能高度依赖状态矩阵的选择与初始化。本文基于SaFARi框架与现有WaLRUS SSMs，提出W4S4（WaLRUS for S4），一种基于冗余小波帧构建的新类SSM。WaLRUS具备稳定对角化能力，支持无需低秩近似的快速核计算，兼具理论严谨性与计算效率。实验表明，相比基于HiPPO的SSMs，WaLRUS在孤立场景及集成至S4等深层架构时，均能显著保留长期信息。在延迟重建、分类基准与长序列建模任务中表现一致提升，验证了小波基状态动态带来的高质量、结构化初始化具有明显优势。该方法为下一代深度SSM模型提供可扩展、多功能的基础。
原文摘要 · Abstract (English)
State Space Models (SSMs) have emerged as powerful components for sequence modeling, enabling efficient handling of long-range dependencies via linear recurrence and convolutional computation. However, their effectiveness depends heavily on the choice and initialization of the state matrix. In this work, we build on the SaFARi framework and existing WaLRUS SSMs to introduce a new variant, W4S4 (WaLRUS for S4), a new class of SSMs constructed from redundant wavelet frames. WaLRUS admits a stable diagonalization and supports fast kernel computation without requiring low-rank approximations, making it both theoretically grounded and computationally efficient. We show that WaLRUS retains information over long horizons significantly better than HiPPO-based SSMs, both in isolation and when integrated into deep architectures such as S4. Our experiments demonstrate consistent improvements across delay reconstruction tasks, classification benchmarks, and long-range sequence modeling, confirming that high-quality, structured initialization enabled by wavelet-based state dynamic offers substantial advantages over existing alternatives. WaLRUS provides a scalable and versatile foundation for the next generation of deep SSM-based models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。