arXiv:2505.00307cs.LG2025-05中稿 · ICML被引 6

用门控机制增强Transformer,同时捕捉时间与变量间依赖,提升多变量时序预测精度。

Gateformer: Advancing Multivariate Time Series Forecasting through Temporal and Variate-Wise Attention with Gated Representations

  • 分变量独立嵌入,再通过门控注意力融合跨时间与跨变量关系。
  • 在13个真实数据集上达领先性能,相较原模型最高提升20.7%。
  • 结构可复用,适合改进现有Transformer或大模型时序预测方法。

近年来,基于Transformer的时序建模受到广泛关注。然而,使用Transformer进行多变量时序预测面临独特挑战:需同时建模时间维度(跨时间)和变量维度(跨变量)的依赖关系。尽管基于Transformer的模型因其灵活性而流行,但如何在架构中有效整合这两种信息源以兼顾性能与效率仍不明确。本文重新设计Transformer架构,有效建模跨时间与跨变量依赖。方法首先对每个变量独立嵌入,生成捕获其时间动态的变量特异性表示,随后在这些嵌入上通过注意力机制建模变量间依赖。在跨时间与跨变量建模阶段引入门控操作,调节信息流动,使模型聚焦于关键特征以实现更准确预测。该方法在13个真实世界数据集上达到最优性能,可无缝集成至其他基于Transformer或大语言模型的预测器中,性能提升最高达20.7%。代码已开源:https://github.com/nyuolab/Gateformer。

原文摘要 · Abstract (English)

There has been a recent surge of interest in time series modeling using the Transformer architecture. However, forecasting multivariate time series with Transformer presents a unique challenge as it requires modeling both temporal (cross-time) and variate (cross-variate) dependencies. While Transformer-based models have gained popularity for their flexibility in capturing both sequential and cross-variate relationships, it is unclear how to best integrate these two sources of information in the context of the Transformer architecture while optimizing for both performance and efficiency. We re-purpose the Transformer architecture to effectively model both cross-time and cross-variate dependencies. Our approach begins by embedding each variate independently into a variate-wise representation that captures its cross-time dynamics, and then models cross-variate dependencies through attention mechanisms on these learned embeddings. Gating operations in both cross-time and cross-variate modeling phases regulate information flow, allowing the model to focus on the most relevant features for accurate predictions. Our method achieves state-of-the-art performance across 13 real-world datasets and can be seamlessly integrated into other Transformer-based and LLM-based forecasters, delivering performance improvements up to 20.7\% over original models. Code is available at this repository: https://github.com/nyuolab/Gateformer.

时序预测Transformer门控机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。