揭示Transformer在时间序列预测中失效的理论原因
Curse of Attention: A Kernel-Based Perspective for Why Transformers Fail to Generalize on Time Series Forecasting and Beyond
- 从核方法视角分析注意力机制的不对称学习问题
- 当前后步符号不一致时，注意力难以捕捉残差特征
- 适合关注模型泛化能力与架构设计的研究者
基于Transformer的时间序列预测（TSF）应用广泛，但许多工作未能超越简单的线性残差模型，其理论机制仍不明确。本文首次提出对Transformer在TSF任务中效率低下的理论解释。我们将其归因于训练过程中注意力网络的{f 不对称学习}现象：当下一步预测任务中前一时刻与当前时刻符号不一致时，注意力无法有效学习残差特征。这导致模型在分布外（OOD）数据上难以泛化，尤其在符号不一致的数据上表现不佳，而线性残差网络则可轻松应对。我们的理论洞察为设计更具表达力和高效性的Transformer架构提供了必要条件。
原文摘要 · Abstract (English)
The application of transformer-based models on time series forecasting (TSF) tasks has long been popular to study. However, many of these works fail to beat the simple linear residual model, and the theoretical understanding of this issue is still limited. In this work, we propose the first theoretical explanation of the inefficiency of transformers on TSF tasks. We attribute the mechanism behind it to {\bf Asymmetric Learning} in training attention networks. When the sign of the previous step is inconsistent with the sign of the current step in the next-step-prediction time series, attention fails to learn the residual features. This makes it difficult to generalize on out-of-distribution (OOD) data, especially on the sign-inconsistent next-step-prediction data, with the same representation pattern, whereas a linear residual network could easily accomplish it. We hope our theoretical insights provide important necessary conditions for designing the expressive and efficient transformer-based architecture for practitioners.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。