arXiv:2411.02941cs.LGcs.AI2024-11被引 18

用Mamba架构实现线性复杂度时序预测,少数据也能高精度。

A Mamba Foundation Model for Time Series Forecasting

  • 基于Mamba的双向编码器捕捉时序依赖,复杂度线性增长。
  • 两阶段迁移学习,少量数据下零样本性能媲美顶尖模型。
  • 适合多变量时序建模,尤其数据稀缺场景下的快速部署。

时序基础模型在零样本学习中表现优异,适用于训练数据稀缺的现实场景。然而,多数模型依赖Transformer架构,输入长度增加时计算复杂度呈二次增长。为此,我们提出TSMamba,一种基于Mamba架构的线性复杂度时序预测基础模型。该模型通过前向与反向Mamba编码器捕捉时序依赖,实现高精度预测。为减少对大数据集的依赖并降低训练成本,TSMamba采用两阶段迁移学习,利用预训练Mamba大语言模型,在中等规模训练集上实现有效建模。第一阶段优化前后向主干网络,通过逐块自回归预测;第二阶段训练预测头并微调其他组件以支持长期预测。尽管主干假设通道独立以适应不同数据集通道数,但在特定多变量数据集微调时引入通道压缩注意力模块,捕捉跨通道依赖。实验表明,尽管训练数据显著减少,TSMamba的零样本性能仍可与现有最佳时序基础模型相当,并在全量数据下达到或超越任务专用模型的性能。代码将公开。

原文摘要 · Abstract (English)

Time series foundation models have demonstrated strong performance in zero-shot learning, making them well-suited for predicting rapidly evolving patterns in real-world applications where relevant training data are scarce. However, most of these models rely on the Transformer architecture, which incurs quadratic complexity as input length increases. To address this, we introduce TSMamba, a linear-complexity foundation model for time series forecasting built on the Mamba architecture. The model captures temporal dependencies through both forward and backward Mamba encoders, achieving high prediction accuracy. To reduce reliance on large datasets and lower training costs, TSMamba employs a two-stage transfer learning process that leverages pretrained Mamba LLMs, allowing effective time series modeling with a moderate training set. In the first stage, the forward and backward backbones are optimized via patch-wise autoregressive prediction; in the second stage, the model trains a prediction head and refines other components for long-term forecasting. While the backbone assumes channel independence to manage varying channel numbers across datasets, a channel-wise compressed attention module is introduced to capture cross-channel dependencies during fine-tuning on specific multivariate datasets. Experiments show that TSMamba's zero-shot performance is comparable to state-of-the-art time series foundation models, despite using significantly less training data. It also achieves competitive or superior full-shot performance compared to task-specific prediction models. The code will be made publicly available.

时序预测Mamba基础模型少样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。