融合长短期时序特征，用跨注意力提升多变量预测精度
S2TX: Cross-Attention Multi-Scale State-Space Transformer for Time Series Forecasting
- 用交叉注意力连接Mamba与Transformer，打通长短期特征
- 在7个基准数据集上达到顶尖性能，内存占用低
- 适合需要高精度多变量时序预测的研究与应用
时间序列预测近年因多尺度模型取得显著进展，用于应对长程与短程模式的异质性。尽管表现优异，我们识别出两个可改进之处：一是多变量时间序列的各变量独立处理；二是长程与短程表示由两个独立模型分别学习，缺乏信息交互。为此，我们提出带交叉注意力的状态空间变压器（S2TX）。S2TX采用交叉注意力机制，将Mamba模型用于提取长程跨变量上下文，同时使用局部窗口注意力的Transformer模型捕捉短程特征。通过跨注意力关注全局上下文，Transformer进一步促进变量间交互及局部与全局信息通信。在七个经典长-短程时间序列预测基准数据集上的全面实验表明，S2TX在保持低内存开销的同时，实现了高度稳健的最先进性能。
原文摘要 · Abstract (English)
Time series forecasting has recently achieved significant progress with multi-scale models to address the heterogeneity between long and short range patterns. Despite their state-of-the-art performance, we identify two potential areas for improvement. First, the variates of the multivariate time series are processed independently. Moreover, the multi-scale (long and short range) representations are learned separately by two independent models without communication. In light of these concerns, we propose State Space Transformer with cross-attention (S2TX). S2TX employs a cross-attention mechanism to integrate a Mamba model for extracting long-range cross-variate context and a Transformer model with local window attention to capture short-range representations. By cross-attending to the global context, the Transformer model further facilitates variate-level interactions as well as local/global communications. Comprehensive experiments on seven classic long-short range time-series forecasting benchmark datasets demonstrate that S2TX can achieve highly robust SOTA results while maintaining a low memory footprint.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。