提出自门控注意力机制,让时间序列预测更快更省内存。
Self-Gating Attention for Efficient Time Series Forecasting

- 用共享矩阵加输入残差表示注意力,避免传统计算方式
- 推理速度和内存消耗变为线性,比标准方法快数倍
- 适合部署在资源受限或高吞吐的实时预测系统
Transformer 在时间序列预测中表现优异,但标准多头自注意力随历史长度呈二次复杂度,限制了其在资源受限或高吞吐场景的应用。我们发现时间序列中的注意力图常存在跨时间戳的冗余模式,源于真实数据中重复的时间模式和相对稳定的时序相关性。为此提出自门控注意力(SGA),以共享可学习矩阵与输入依赖残差项表示注意力分数,避免查询与键的投影操作,使计算时间和得分矩阵内存复杂度均降至线性。将 SGA 集成至多个预测主干,在九个公开数据集(涵盖电力、金融、气象、医疗监测、人体活动、气候等)上测试,结果表明其在保持与先进注意力机制相当预测性能的同时,显著提升推理效率,为实际部署提供了有力证据。
原文摘要 · Abstract (English)
Transformer architectures have shown strong potential in time series forecasting, where multi-head self-attention is widely used to capture temporal dependencies across historical timestamps. However, standard self-attention has quadratic time and memory complexity with respect to the look-back length. This cost may limit its use in resource-constrained or high-throughput forecasting systems, where fast and memory-efficient inference is important. Through qualitative and quantitative analyses, we observe that self-attention maps in time series forecasting often contain redundant patterns across different timestamps. This phenomenon can be related to the repeated temporal patterns and relatively stable temporal correlations in many real-world time series. Motivated by this observation, we propose Self-Gating Attention (SGA), a plug-and-play attention mechanism that represents the attention score with a shared learnable matrix and an input-dependent residual component. The shared matrix captures common attention patterns, while the residual component captures input-dependent variations. In this way, SGA avoids the query and key projections used in standard attention score computation, leading to linear time and score-matrix memory complexity with respect to the look-back length. We integrate SGA into several forecasting backbones and compare it with standard self-attention and lightweight attention variants on nine publicly available real-world datasets covering electricity, finance, weather, medical monitoring, human activity, and climate records. The results show that SGA improves inference efficiency on public benchmarks while maintaining competitive forecasting performance against state-of-the-art attention mechanisms. These benchmark results provide deployment-oriented evidence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。