提出局部注意力机制，让Transformer更快处理长时序数据
Local Attention Mechanism: Boosting the Transformer Architecture for Long-Sequence Time Series Forecasting
- 用时间序列连续性减少计算量，仅需O(nlogn)复杂度
- 在多个长序列预测任务上超越现有模型性能
- 适合需要高效处理长时间序列的工业场景
Transformer已成为自然语言处理的主流架构，并逐步应用于时间序列分析，尤其在长时序预测中表现出色。本文提出局部注意力机制（LAM），利用时间序列的连续特性，显著降低注意力计算量。我们设计了一种基于张量代数的LAM实现算法，时间与内存复杂度均为O(nlogn)，远优于传统方法的O(n²)。此外，针对现有评估数据集不足的问题，我们构建了新的长时序预测数据集。实验表明，引入LAM的Transformer在多个基准任务上超越当前最优模型，验证了该方法的有效性，并揭示了长序列预测领域未来面临的挑战。
原文摘要 · Abstract (English)
Transformers have become the leading choice in natural language processing over other deep learning architectures. This trend has also permeated the field of time series analysis, especially for long-horizon forecasting, showcasing promising results both in performance and running time. In this paper, we introduce Local Attention Mechanism (LAM), an efficient attention mechanism tailored for time series analysis. This mechanism exploits the continuity properties of time series to reduce the number of attention scores computed. We present an algorithm for implementing LAM in tensor algebra that runs in time and memory O(nlogn), significantly improving upon the O(n^2) time and memory complexity of traditional attention mechanisms. We also note the lack of proper datasets to evaluate long-horizon forecast models. Thus, we propose a novel set of datasets to improve the evaluation of models addressing long-horizon forecasting challenges. Our experimental analysis demonstrates that the vanilla transformer architecture magnified with LAM surpasses state-of-the-art models, including the vanilla attention mechanism. These results confirm the effectiveness of our approach and highlight a range of future challenges in long-sequence time series forecasting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。