arXiv:2501.13989cs.LGcs.AI2025-01IJCAI被引 47

通过频域增强注意力机制,提升多变量时间序列预测精度。

FreEformer: Frequency Enhanced Transformer for Multivariate Time Series Forecasting

  • 将时间序列转至频域,用Transformer捕捉跨变量依赖
  • 频域注意力矩阵经改进后,在18个真实数据集上全面领先
  • 适合需要高精度时序建模的电力、交通与金融场景

本文提出FreEformer,一种基于频率增强的Transformer模型,用于多变量时间序列预测。该方法假设频谱能提供全局视角,有助于鲁棒表征学习。首先利用离散傅里叶变换(DFT)将时间序列转换至复数频域,随后在频谱上应用Transformer,分别处理实部与虚部。然而,原始注意力矩阵呈现低秩特性,限制了表征多样性,这可能源于频域固有的稀疏性及Softmax对高值的聚焦。为此,我们在原注意力矩阵基础上引入可学习矩阵,并进行逐行L1归一化。理论分析表明,该机制提升了特征多样性和梯度流动。大量实验显示,FreEformer在涵盖电力、交通、气象、医疗和金融的18个真实基准上持续优于现有先进模型。值得注意的是,该增强注意力机制亦能提升现有基于Transformer的预测器性能。

原文摘要 · Abstract (English)

This paper presents \textbf{FreEformer}, a simple yet effective model that leverages a \textbf{Fre}quency \textbf{E}nhanced Trans\textbf{former} for multivariate time series forecasting. Our work is based on the assumption that the frequency spectrum provides a global perspective on the composition of series across various frequencies and is highly suitable for robust representation learning. Specifically, we first convert time series into the complex frequency domain using the Discrete Fourier Transform (DFT). The Transformer architecture is then applied to the frequency spectra to capture cross-variate dependencies, with the real and imaginary parts processed independently. However, we observe that the vanilla attention matrix exhibits a low-rank characteristic, thus limiting representation diversity. This could be attributed to the inherent sparsity of the frequency domain and the strong-value-focused nature of Softmax in vanilla attention. To address this, we enhance the vanilla attention mechanism by introducing an additional learnable matrix to the original attention matrix, followed by row-wise L1 normalization. Theoretical analysis~demonstrates that this enhanced attention mechanism improves both feature diversity and gradient flow. Extensive experiments demonstrate that FreEformer consistently outperforms state-of-the-art models on eighteen real-world benchmarks covering electricity, traffic, weather, healthcare and finance. Notably, the enhanced attention mechanism also consistently improves the performance of state-of-the-art Transformer-based forecasters.

时间序列频域建模Transformer预测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。