让注意力机制学会处理振荡信号,提升时序建模能力
Beyond Similarity: Temporal Operator Attention for Time Series Analysis

- 用可学习的时序算子替代传统注意力的加权混合
- 在重建任务中性能提升显著,最高增益达12.3%
- 适合需要精确捕捉周期性与波动特性的时序场景
时序预测中一个长期存在的悖论是:结构简单的MLP和线性模型常优于高容量的Transformer。我们指出,这一差距源于序列建模基础范式的不匹配:许多时序动态由全局时序算子(如滤波和谐波结构)主导,而标准注意力将每个输出视为输入的凸组合,限制了对带符号和振荡变换的表达能力。我们将此限制形式化为softmax注意力中的单纯形约束混合瓶颈,尤其在算子驱动的时序任务中尤为突出。为此,我们提出时空算子注意力(Temporal Operator Attention, TOA),通过引入显式可学习的序列空间算子,实现跨时间的直接带符号混合,同时保持输入依赖的自适应性。为使密集的$N imes N$算子实用,我们引入随机算子正则化(Stochastic Operator Regularization),一种高方差丢弃机制,稳定训练并防止平凡记忆。在预测、异常检测和分类基准上,将TOA集成到PatchTST和iTransformer等主流骨干网络中,均取得持续性能提升,尤其在重构密集型任务中表现突出。结果表明,显式算子学习是高效时序建模的关键要素。
原文摘要 · Abstract (English)
A persistent paradox in time-series forecasting is that structurally simple MLP and linear models often outperform high-capacity Transformers. We argue that this gap arises from a mismatch in the sequence-modeling primitive: while many time-series dynamics are governed by global temporal operators (e.g., filtering and harmonic structure), standard attention forms each output as a convex combination of inputs. This restricts its ability to represent signed and oscillatory transformations that are fundamental to temporal signal processing. We formalize this limitation as a simplex-constrained mixing bottleneck in softmax attention, which becomes especially restrictive for operator-driven time-series tasks. To address this, we propose $\textbf{Temporal Operator Attention (TOA)}$, a framework that augments attention with explicit, learnable sequence-space operators, enabling direct signed mixing across time while preserving input-dependent adaptivity. To make dense $N \times N$ operators practical, we introduce Stochastic Operator Regularization, a high-variance dropout mechanism that stabilizes training and prevents trivial memorization. Across forecasting, anomaly detection, and classification benchmarks, TOA consistently improves performance when integrated into standard backbones such as PatchTST and iTransformer, with particularly strong gains in reconstruction-heavy tasks. These results suggest that explicit operator learning is a key ingredient for effective time-series modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。