将线性Transformer重构为可解释的动态VAR模型，提升时间序列预测性能与可解释性
Linear Transformers as VAR Models: Aligning Autoregressive Attention Mechanisms with Autoregressive Forecasting
- 用动态VAR结构重解单层线性注意力机制
- 多层架构通过结构调整实现与自回归目标对齐
- 适合关注可解释性与效率的时间序列研究者
自回归注意力机制在时间序列预测中备受关注，其中线性注意力有时优于标准注意力。然而，深层Transformer架构常与自回归目标不匹配，掩盖了线性注意力中隐含的向量自回归（VAR）结构，削弱了对数据生成过程的建模能力。本文首先证明单层线性注意力可被解释为动态向量自回归（VAR）结构；随后指出现有多层Transformer存在结构偏差，影响可解释性与泛化能力。为此，我们提出通过重新排列MLP、注意力与输入输出流程，使多层线性注意力也能对齐为VAR模型。进而提出结构对齐的动态VAR混合模型（SAMoVAR），融合可解释的动态VAR权重，用于多变量时间序列预测。通过与自回归目标对齐架构，SAMoVAR在性能、可解释性与计算效率上均优于当前最优模型。
原文摘要 · Abstract (English)
Autoregressive attention-based time series forecasting (TSF) has drawn increasing interest, with mechanisms like linear attention sometimes outperforming vanilla attention. However, deeper Transformer architectures frequently misalign with autoregressive objectives, obscuring the underlying VAR structure embedded within linear attention and hindering their ability to capture the data generative processes in TSF. In this work, we first show that a single linear attention layer can be interpreted as a dynamic vector autoregressive (VAR) structure. We then explain that existing multi-layer Transformers have structural mismatches with the autoregressive forecasting objective, which impair interpretability and generalization ability. To address this, we show that by rearranging the MLP, attention, and input-output flow, multi-layer linear attention can also be aligned as a VAR model. Then, we propose Structural Aligned Mixture of VAR (SAMoVAR), a linear Transformer variant that integrates interpretable dynamic VAR weights for multivariate TSF. By aligning the Transformer architecture with autoregressive objectives, SAMoVAR delivers improved performance, interpretability, and computational efficiency, comparing to SOTA TSF models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。