用卷积提取局部特征,再用注意力建模全局依赖,提升长序列时间预测效果。
Patch-Level Tokenization with CNN Encoders and Attention for Improved Transformer Time-Series Forecasting
- 先用卷积处理固定长度片段,提取局部动态特征
- 在扩展序列长度下,性能优于卷积基线模型
- 适合需要长时序建模的多变量时间序列任务
基于Transformer的模型在时间序列预测中表现优异,依赖自注意力机制捕捉长期时序依赖。然而其效果高度依赖于原始多变量时间序列数据的输入表示质量,尤其在序列变长、数据规模增大时更为关键。本文提出一种两阶段预测框架,将局部时序表征学习与全局依赖建模明确分离:首先利用卷积神经网络对固定长度的时间片段进行处理,提取短程时序动态和非线性特征交互,生成紧凑的片段级令牌嵌入;随后在表征学习阶段应用令牌级自注意力以优化嵌入,再由Transformer编码器建模片段间时序依赖,生成预测结果。该方法在包含受控静态与动态因子的合成多变量时间序列数据集上评估,采用更长序列长度和更多样本。实验表明,在增加时序上下文条件下,所提框架持续优于卷积基线模型,并保持与强基准片段化Transformer模型相当的竞争力。结果表明,结构化的片段级令牌化能为多变量时间序列预测提供可扩展且高效的表示,尤其适用于较长输入序列场景。
原文摘要 · Abstract (English)
Transformer-based models have shown strong performance in time-series forecasting by leveraging self-attention to model long-range temporal dependencies. However, their effectiveness depends critically on the quality and structure of input representations derived from raw multivariate time-series data, particularly as sequence length and data scale increase. This paper proposes a two-stage forecasting framework that explicitly separates local temporal representation learning from global dependency modelling. In the proposed approach, a convolutional neural network operates on fixed-length temporal patches to extract short-range temporal dynamics and non-linear feature interactions, producing compact patch-level token embeddings. Token-level self-attention is applied during representation learning to refine these embeddings, after which a Transformer encoder models inter-patch temporal dependencies to generate forecasts. The method is evaluated on a synthetic multivariate time-series dataset with controlled static and dynamic factors, using an extended sequence length and a larger number of samples. Experimental results demonstrate that the proposed framework consistently outperforms a convolutional baseline under increased temporal context and remains competitive with a strong patch-based Transformer model. These findings indicate that structured patch-level tokenization provides a scalable and effective representation for multivariate time-series forecasting, particularly when longer input sequences are considered.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。