提出轻量Transformer LATST，解决时序预测中注意力失效问题
LATST: Are Transformers Necessarily Complex for Time-Series Forecasting
- 设计新架构缓解注意力熵崩溃与训练不稳
- 在多数据集上超越现有SOTA Transformer模型
- 参数更少却性能接近线性模型，适合资源受限场景
基于Transformer的架构在自然语言处理和计算机视觉领域取得了显著成功，但在多变量长期时序预测任务中的表现往往不及简单的线性基线模型。先前研究指出，传统的注意力机制是制约其效果的关键因素。为此，本文提出LATST，一种旨在缓解熵崩溃和训练不稳定性等常见挑战的新方法。我们在多个真实世界多变量时间序列数据集上对LATST进行了严格评估，结果表明其能够超越现有的最先进Transformer模型。值得注意的是，LATST在某些数据集上的表现与部分线性模型相当，但所需参数更少，展现出更高的效率与有效性。
原文摘要 · Abstract (English)
Transformer-based architectures have achieved remarkable success in natural language processing and computer vision. However, their performance in multivariate long-term forecasting often falls short compared to simpler linear baselines. Previous research has identified the traditional attention mechanism as a key factor limiting their effectiveness in this domain. To bridge this gap, we introduce LATST, a novel approach designed to mitigate entropy collapse and training instability common challenges in Transformer-based time series forecasting. We rigorously evaluate LATST across multiple real-world multivariate time series datasets, demonstrating its ability to outperform existing state-of-the-art Transformer models. Notably, LATST manages to achieve competitive performance with fewer parameters than some linear models on certain datasets, highlighting its efficiency and effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。