arXiv:2605.08539cs.LGcs.AI2026-05

探究时序模型的连续性,发现连续行为能提升性能。

Continuity Laws for Sequential Models

论文配图:Continuity Laws for Sequential Models
图 1 · 摘自论文原文
  • 以时间细化下的收敛性定义模型连续性
  • S4具稳定连续性,S6对输入敏感且不连续
  • 连续性可指导高效采样,适合时序建模任务

归纳偏差影响时序模型的行为与性能。本文研究一种被忽视的归纳偏差:时间连续性。我们提出:基于连续时间形式的模型(如状态空间模型)是否真的在时间上表现连续?这是否带来对具有连续时间结构任务的性能提升?为此,我们形式化了模型连续性——当时间离散化细化时,模型预测趋近于一个潜在的连续轨迹。结果表明,S4表现出稳定的连续行为,而尽管源自连续动力系统,S6(Mamba核心)却对输入幅度和选择性动态更敏感。为评估其学习意义,我们进一步引入直接从时间结构量化数据集连续性的指标。在多个基准上,任务连续性、模型连续性与模型性能呈现明确对齐。连续性不仅是归纳偏好,还具实际价值:我们证明它支持一种简单的时间子采样策略,提升效率与性能。

原文摘要 · Abstract (English)

Inductive biases influence the behavior and performance of sequential models. In this work, we study an underexplored inductive bias in sequential modeling: continuity in time. We ask a simple question: do models motivated by continuous-time formulations, such as state-space models, actually behave continuously in time, and does this translate into better performance on tasks with continuous temporal structure? To answer this, we formalize model continuity as convergence under temporal refinement, where a model is continuous if its predictions approach an underlying continuous trajectory as the temporal discretization is refined. We show that S4 exhibits stable continuous behavior, whereas S6 (the core of Mamba) can be more sensitive to input amplitude and selective dynamics, despite being derived from a continuous dynamical system. To study whether this distinction matters for learning, we also need a corresponding notion of task continuity. We therefore introduce a metric to quantify the continuity of datasets directly from their temporal structure. Across benchmarks, we find a clear empirical alignment between task continuity, model continuity, and model performance. Beyond an inductive bias, continuity also has practical consequences: we show that it enables a simple temporal subsampling strategy that improves both efficiency and performance.

时序建模连续性S4S6

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。