提出首个通用时间序列模型GTM,提升多任务表征学习能力。
GTM: A General Time-series Model for Enhanced Representation Learning of Time-Series Data
- 通过频域注意力捕捉时间粒度特征,增强表征学习
- 混合掩码预训练策略使生成任务表现超越现有模型
- 无需修改即可适配多种生成任务,适合多场景应用
尽管时间序列基础模型取得进展,但在提升表征学习和适应多样化下游任务方面仍存挑战。本文提出通用时间序列模型(GTM),通过新颖的频域注意力机制捕捉时间粒度感知特征,该方面在以往研究中未被充分探索。我们进一步设计一种新型预训练策略,利用混合掩码机制统一重构与自回归目标。结合2D位置编码与跨度打乱,提升了表示的鲁棒性与泛化能力。GTM是首个生成任务无关的时间序列分析模型,可无须任务特定调整即可无缝适配各类生成任务。大量实验表明,GTM在多种生成任务上持续优于当前最优模型,并在分类任务中仅需极少调整即达强性能。此外,随着模型规模和预训练数据增加,其准确率呈现明显上升趋势。
原文摘要 · Abstract (English)
Despite recent progress in time-series foundation models, challenges persist in improving representation learning and adapting to diverse downstream tasks. We introduce a General Time-series Model (GTM), which advances representation learning via a novel frequency-domain attention mechanism that captures time-granularity-aware features, an aspect underexplored in prior research. We further propose a novel pre-training strategy that unifies reconstruction and autoregressive objectives through a hybrid masking mechanism. Our pre-training strategy, combined with 2D positional encoding and span shuffling, enhances the robustness and generalization of representations. GTM is established as the first generative-task-agnostic model for time-series analysis, enabling seamless adaptation to various generative tasks without any task-specific modifications. Extensive experiments demonstrate that GTM consistently outperforms SOTA models on various generative tasks and achieves strong classification results with minimal adaptation. Furthermore, GTM exhibits clear scaling behavior, with accuracy improving as model size and pre-training data increase.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。