arXiv:2504.08964cs.LG2025-04被引 2

BLUR模型用双向线性RNN高效建模长序列依赖,性能超越Transformer。

Bidirectional Linear Recurrent Models for Sequence-Level Multisource Fusion

  • 采用前向后向线性递归单元,同时捕捉历史与未来信息。
  • 在时间序列数据上准确率超Transformer,计算成本降低显著。
  • 适合需高效处理长序列的现实预测任务,如气象与能源负荷。

序列建模是广泛应用于天气预测、温度监测和能源负荷预测等领域的关键挑战任务。尽管基于注意力机制的Transformer因并行训练效率高成为当前主流,但其时间复杂度为二次方,限制了长序列的可扩展性。相比之下,循环神经网络(RNN)具有线性时间复杂度,推动了线性RNN的复兴。本文提出BLUR(Bidirectional Linear Unit for Recurrent network),通过前向与后向线性递归单元(LRUs)高效捕获过去与未来的依赖关系。BLUR保持传统RNN的线性时间复杂度,同时通过LRU实现快速并行训练。此外,它具备可证明的训练稳定性与强大的近似能力,能有效建模长期依赖。在图像序列与时间序列数据集上的大量实验表明,BLUR不仅在准确性上优于Transformer与传统RNN,还显著降低计算开销,特别适用于真实世界中的预测任务。代码已公开。

原文摘要 · Abstract (English)

Sequence modeling is a critical yet challenging task with wide-ranging applications, especially in time series forecasting for domains like weather prediction, temperature monitoring, and energy load forecasting. Transformers, with their attention mechanism, have emerged as state-of-the-art due to their efficient parallel training, but they suffer from quadratic time complexity, limiting their scalability for long sequences. In contrast, recurrent neural networks (RNNs) offer linear time complexity, spurring renewed interest in linear RNNs for more computationally efficient sequence modeling. In this work, we introduce BLUR (Bidirectional Linear Unit for Recurrent network), which uses forward and backward linear recurrent units (LRUs) to capture both past and future dependencies with high computational efficiency. BLUR maintains the linear time complexity of traditional RNNs, while enabling fast parallel training through LRUs. Furthermore, it offers provably stable training and strong approximation capabilities, making it highly effective for modeling long-term dependencies. Extensive experiments on sequential image and time series datasets reveal that BLUR not only surpasses transformers and traditional RNNs in accuracy but also significantly reduces computational costs, making it particularly suitable for real-world forecasting tasks. Our code is available here.

序列建模线性RNN双向建模长序列

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。