用MLP替代注意力机制，大幅降低计算量仍保持高性能。
RAM: Replace Attention with MLP for Efficient Multivariate Time Series Forecasting
- 用前馈层和残差连接替代注意力计算流程
- 时空模型减62.5%计算量，长时序模型减42.2%且性能下降<2.5%
- 适合追求高效推理的工业级时间序列预测场景
基于注意力的架构在多变量时间序列预测中广泛应用，包括时空预测（STF）和长期预测（LTSF）。然而我们对其有效性机制仍理解有限。本文提出一种新剪枝策略——将注意力替换为MLP（RAM），仅通过前馈层、残差连接和层归一化即可实现时间与空间建模。具体地，可移除注意力中的查询、键、值投影、注意力分数计算、点积加权及最终投影等环节，而性能几乎不受影响。该方法在时空模型上实现62.579%的浮点运算量（FLOPs）降低，性能下降不足2.5%；在长时序模型上则减少42.233%的计算量，性能损失低于2%。所提模型仍保持与当前最优方法相当的性能水平。
原文摘要 · Abstract (English)
Attention-based architectures have become ubiquitous in time series forecasting tasks, including spatio-temporal (STF) and long-term time series forecasting (LTSF). Yet, our understanding of the reasons for their effectiveness remains limited. In this work, we propose a novel pruning strategy, $\textbf{R}$eplace $\textbf{A}$ttention with $\textbf{M}$LP (RAM), that approximates the attention mechanism using only feedforward layers, residual connections, and layer normalization for temporal and/or spatial modeling in multivariate time series forecasting. Specifically, the Q, K, and V projections, the attention score calculation, the dot-product between the attention score and the V, and the final projection can be removed from the attention-based networks without significantly degrading the performance, so that the given network remains the top-tier compared to other SOTA methods. RAM achieves a $62.579\%$ reduction in FLOPs for spatio-temporal models with less than $2.5\%$ performance drop, and a $42.233\%$ FLOPs reduction for LTSF models with less than $2\%$ performance drop.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。