对比Transformer架构,发现双向注意力+直接映射最适长期时间序列预测。
The Power of Architecture: Deep Dive into Transformer Architectures for Long-Term Time Series Forecasting
- 提出新分类法,分离架构与特定设计,可公平比较不同Transformer结构。
- 双向联合注意力与完整聚合策略显著提升预测准确率,优于自回归方法。
- 适合关注时序建模架构设计的研究者,尤其长周期预测场景。
基于Transformer的模型在长期时间序列预测(LTSF)中占据主导地位,但编码器仅、编码器-解码器和解码器仅等架构差异引发关键问题:哪种架构最适合LTSF?现有模型常与特定时序设计深度耦合,难以剥离架构本身的影响。为此,我们提出一种新分类体系,解耦各类设计,实现对Transformer架构更清晰、统一的比较。该分类涵盖注意力机制、预测聚合方式、预测范式及归一化层等核心方面。通过大量实验,我们发现:双向注意力结合联合注意力最有效;更完整的预测聚合策略性能更优;直接映射范式优于自回归方法。基于最优架构组合的模型持续超越多个现有模型,验证了结论有效性。代码已开源:https://github.com/HALF111/TSF_architecture。
原文摘要 · Abstract (English)
Transformer-based models have recently become dominant in Long-term Time Series Forecasting (LTSF), yet the variations in their architecture, such as encoder-only, encoder-decoder, and decoder-only designs, raise a crucial question: What Transformer architecture works best for LTSF tasks? However, existing models are often tightly coupled with various time-series-specific designs, making it difficult to isolate the impact of the architecture itself. To address this, we propose a novel taxonomy that disentangles these designs, enabling clearer and more unified comparisons of Transformer architectures. Our taxonomy considers key aspects such as attention mechanisms, forecasting aggregations, forecasting paradigms, and normalization layers. Through extensive experiments, we uncover several key insights: bi-directional attention with joint-attention is most effective; more complete forecasting aggregation improves performance; and the direct-mapping paradigm outperforms autoregressive approaches. Furthermore, our combined model, utilizing optimal architectural choices, consistently outperforms several existing models, reinforcing the validity of our conclusions. We hope these findings offer valuable guidance for future research on Transformer architectural designs in LTSF. Our code is available at https://github.com/HALF111/TSF_architecture.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。