揭示时间序列Transformer为何失效，发现其实际退化为MLP
Why Attention Fails: The Degeneration of Transformers into MLPs in Time Series Forecasting
- 逐步将Transformer改造为MLP，验证注意力机制失效
- 实验证明现有时间序列Transformer多已退化为简单MLP
- 指出嵌入方法缺陷是导致注意力失效的根本原因
基于Transformer的模型在自然语言处理和计算机视觉中表现优异，但在时间序列预测任务中却未展现出明显优势，甚至在某些情况下表现不如简单的线性基线模型。然而，大多数研究尚未深入探究Transformer失败的原因。为更好地理解时间序列Transformer（TST），我们设计了一系列实验，逐步将Transformer修改为MLP，以研究注意力机制的影响。令人惊讶的是，现有时间序列Transformer中的注意力模块往往退化为简单的MLP。为此，我们构建了一个可解释的数据集，探究注意力机制失效的原因，发现其并未按预期工作。我们从理论上分析了这一现象的根源，证明当前的嵌入方法无法使Transformer在结构良好的潜在空间中运作，并进一步剖析了嵌入方法失效的深层原因。
原文摘要 · Abstract (English)
Transformer-based architectures achieved high performance in natural language processing and computer vision, yet many studies have shown that they have not demonstrated a clear advantage in time series forecasting and even underperform simple linear baselines in some cases. However, most of these studies have not thoroughly explored the reasons behind the failure of transformers. To better understand time-series transformers(TST), we designed a series of experiments, progressively modifying transformers into MLPs to investigate the impact of the attention mechanism. Surprisingly, transformer blocks often degenerate into simple MLPs in existing time-series transformers. We designed a interpretable dataset to investigate the reasons behind the failure of the attention mechanism and revealed that the attention mechanism is not working in the expected way. We theoretically analyzed the reasons behind this phenomenon, demonstrating that the current embedding methods fail to allow transformers to function in a well-structured latent space, and further analyzed the deeper underlying causes of the failure of embedding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。