提出连续时间注意力机制,让模型更稳定地处理不规则长序列数据。
FLUID: Continuous-Time Hyperconnected Sparse Transformer for Sink-Free Learning

- 用微分方程重定义注意力,实现连续动态建模
- 在多个任务中提升47%性能,增强抗噪与泛化能力
- 适合长序列、稀疏数据、自动驾驶等复杂场景
连续时间(CT)Transformer通过利用输入或输出嵌入的连续动态,提升了对不规则和长距离序列的建模能力,但其核心缩放点积注意力(SDPA)机制仍为离散。本文提出FLUID(灵活统一信息动力学),将连续动态直接引入注意力计算,以液态注意力网络(LAN)替代传统注意力。LAN将注意力得分重新解释为依赖输入的非线性递归门调制的线性常微分方程解,理论上保证了动力学稳定性,并在特定参数下可退化为SDPA或CT-RNN。LAN还引入显式注意力-汇点门,消除对无信息节点的过度关注。FLUID用输入依赖的液态超连接替代标准残差连接,自适应调节层间信息流。实验涵盖不规则时间序列、长程建模、自动驾驶车道保持控制及稀缺数据下的物理动力学学习。结果表明,FLUID在所有任务中均匹配或超越基准,部分场景提升达47%,显著提升分布外泛化能力,且具备更强噪声鲁棒性与自校正归纳偏置。我们还分析关键超参数,发现其在运行时与内存效率上处于竞争方法的中间位置。
原文摘要 · Abstract (English)
Continuous-time (CT) Transformers improve irregular and long-range modeling over CT-RNNs by exploiting inputs or outputs embeddings with continuous dynamics. However, the core scaled-dot-product-attention (SDPA) mechanism remains inherently discrete. We propose FLUID (Flexible Unified Information Dynamics), a CT Transformer that incorporates continuous dynamics directly into the attention computation by replacing it with Liquid Attention Network (LAN). LAN reinterprets attention logits as continuous dynamical system and reformulates them as the solution to a linear ODE modulated by input-dependent nonlinear recurrent gates. Theoretically, we establish stability guarantees for LAN dynamics and show that it serves as an interpolating middle ground between SDPA and CT-RNNs, recovering each as special case under well-defined parameterization of its gating functions. LAN also introduces an explicit attention-sink gate to eliminate disproportionate attention mass on uninformative nodes. FLUID replaces standard residual connections with input-dependent Liquid Hyper-Connections to adaptively regulate interlayer information flow. Empirically, we evaluate FLUID on a broad set of learning tasks, including (i) irregular time-series, (ii) long-range modeling, (iii) lane-keeping control of autonomous vehicles, and (iv) learning physical dynamics under a scarce data regime. Across all the tasks, FLUID consistently matches or outperforms CT baselines, achieving improvements of up to 47% in certain scenarios and enhancing generalization under distributional shifts. Additionally, FLUID demonstrates superior noise robustness and a self-correcting inductive bias in autonomous vehicle control. We also provide a detailed analysis of key hyperparameters to guide tuning and show that FLUID occupies an intermediate position among competing approaches in terms of runtime and memory efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。