通过深度循环与输入重塑,提升时序分类中SSM的性能。
Looped SSMs: Depth-Recurrence and Input Reshaping for Time Series Classification
- 采用深度循环结构,重复使用相同参数块以简化优化。
- 在6个基准上提升1-6%准确率,优于独立参数模型。
- 适合关注模型效率与设计创新的时序分析研究者。
状态空间模型(SSMs)在序列维度上具有固有的递归性,但深度递归——即在多层中重复使用相同模块——在该模型家族中尚未被探索。我们发现,将具有 $k$ 个参数的循环式SSM迭代 $L$ 次,其表现始终接近或优于具有 $k \cdot L$ 个独立参数的标准SSM,涵盖四种架构(LRU、S5、LinOSS、LrcSSM)和六个时间序列分类基准。尽管假设空间更小,这一优势无法用表达能力解释,反而表明跨深度参数共享是一种有益的归纳偏置,有助于优化。此外,我们发现输入重塑是被忽视的设计维度:对低维输入拼接时间步,或对高维输入展开并重分块特征-时间维度,均带来1-6%的准确率提升,且在5个随机种子下稳定验证。两项技术各自独立有效,组合后效果叠加,说明深度递归与输入重塑是时间序列中SSM的两个独立且未被充分探索的设计方向。
原文摘要 · Abstract (English)
State Space Models (SSMs) are inherently recurrent along the sequence dimension, yet depth-recurrence - reusing the same block repeatedly across layers, as recently applied in looped transformers - has not been explored in this model family. We show that a looped SSM with $k$ parameters iterated $L$ times consistently closely matches or outperforms a standard SSM with $k \cdot L$ independent parameters across four architectures (LRU, S5, LinOSS, LrcSSM) and six time series classification benchmarks, despite operating within a strictly smaller hypothesis space, as we formally establish. Since the larger model contains the looped model as a special case, this dominance cannot be explained by expressivity and instead points to parameter sharing across depth as a beneficial inductive bias that simplifies optimization. These results demonstrate that depth-recurrence is orthogonal to sequence-recurrence and independently beneficial. We further show that input reshaping is an equally neglected design axis: concatenating timesteps for low-dimensional inputs, or flattening and rechunking the joint feature-time dimension for high-dimensional ones, yields accuracy gains of 1-6% across all models, confirmed over 5 random seeds. Both techniques provide standalone improvements that compound when combined, suggesting that depth and input reshaping are two independent and underexplored design axes for SSMs on time series.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。