S7模型通过动态调整状态转移，高效处理长序列输入依赖问题。
S7: Selective and Simplified State Space Layers for Sequence Modeling
- 基于输入内容动态调整状态转移，实现输入依赖建模。
- 在长序列任务中超越多个基线模型，包括神经形态事件数据集和物理时间序列。
- 无需复杂领域先验，设计简洁但性能显著提升，适合长序列建模场景。
序列建模的核心挑战在于高效处理长上下文任务。尽管近期状态空间模型（SSMs）在此方面取得进展，但往往缺乏输入依赖性过滤机制，或需大幅增加模型复杂度以应对输入变化。本文提出S7，一种简化而强大的SSM，可在保持效率的同时实现输入依赖建模。S7采用稳定重参数化与特定设计，使状态转移动态响应输入内容，确保长期序列建模中状态演化稳定。该重参数化还控制梯度范数，实现高效训练并防止梯度爆炸或消失。S7在多种序列建模任务中显著优于基线模型，涵盖神经形态事件数据集、Long Range Arena基准及各类物理与生物时间序列。整体而言，S7提供了一种无需依赖复杂领域先验的更简洁序列建模方法，在关键基准上实现显著性能提升。
原文摘要 · Abstract (English)
A central challenge in sequence modeling is efficiently handling tasks with extended contexts. While recent state-space models (SSMs) have made significant progress in this area, they often lack input-dependent filtering or require substantial increases in model complexity to handle input variability. We address this gap by introducing S7, a simplified yet powerful SSM that can handle input dependence while incorporating stable reparameterization and specific design choices to dynamically adjust state transitions based on input content, maintaining efficiency and performance. We prove that this reparameterization ensures stability in long-sequence modeling by keeping state transitions well-behaved over time. Additionally, it controls the gradient norm, enabling efficient training and preventing issues like exploding or vanishing gradients. S7 significantly outperforms baselines across various sequence modeling tasks, including neuromorphic event-based datasets, Long Range Arena benchmarks, and various physical and biological time series. Overall, S7 offers a more straightforward approach to sequence modeling without relying on complex, domain-specific inductive biases, achieving significant improvements across key benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。