arXiv:2511.03190cs.LGcs.AI2025-11

提出线性注意力机制,提升多变量时序建模效率。

Efficient Linear Attention for Multivariate Time Series Modeling via Entropy Equality

  • 基于熵相等原理设计线性复杂度注意力,避免传统方法的二次开销。
  • 在四个时空数据集上实现更优预测性能,内存与计算时间显著降低。
  • 适合长序列时序建模任务,尤其关注效率与可扩展性的研究者。

注意力机制因其捕捉复杂依赖关系的能力,在各类应用中广泛应用,包括时序建模;然而其使用常受限于二次计算复杂度,制约了长序列的可扩展性。本文提出一种新型线性注意力机制,以克服这些限制。该方法基于理论证明:熵作为概率单纯形上的严格凹函数,表明具有相同概率排序和相似熵值的分布具有结构相似性。基于此洞察,我们开发了一种高效近似算法,仅以线性复杂度计算点积生成分布的熵,从而实现基于熵相等的线性注意力机制。通过严谨分析,我们发现注意力在时空时序建模中的有效性,可能并非主要源于softmax的非线性,而是源于达到适度且均衡的权重分布。在四个时空数据集上的大量实验验证了该方法,表现出竞争性或更优的预测性能,同时大幅降低内存占用和计算时间。

原文摘要 · Abstract (English)

Attention mechanisms have been extensively employed in various applications, including time series modeling, owing to their capacity to capture intricate dependencies; however, their utility is often constrained by quadratic computational complexity, which impedes scalability for long sequences. In this work, we propose a novel linear attention mechanism designed to overcome these limitations. Our approach is grounded in a theoretical demonstration that entropy, as a strictly concave function on the probability simplex, implies that distributions with aligned probability rankings and similar entropy values exhibit structural resemblance. Building on this insight, we develop an efficient approximation algorithm that computes the entropy of dot-product-derived distributions with only linear complexity, enabling the implementation of a linear attention mechanism based on entropy equality. Through rigorous analysis, we reveal that the effectiveness of attention in spatio-temporal time series modeling may not primarily stem from the non-linearity of softmax but rather from the attainment of a moderate and well-balanced weight distribution. Extensive experiments on four spatio-temporal datasets validate our method, demonstrating competitive or superior forecasting performance while achieving substantial reductions in both memory usage and computational time.

注意力机制时序建模线性复杂度熵相等

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。