用自蒸馏方法提升时间序列分类模型的预训练效果
UTICA: Multi-Objective Self-Distllation Foundation Model Pretraining for Time Series Classification
- 采用学生-教师框架，结合裁剪与掩码学习时序不变性与局部结构
- 在UCR和UEA基准上达到当前最优分类性能
- 适合做时间序列建模的开发者参考
自监督基础模型在多个领域取得显著进展，包括时间序列。然而，非对比性方法——这一在计算机视觉中推动重要进展的范式——在时间序列领域的潜力尚未被充分探索。本文将DINOv2风格的自蒸馏方法应用于时间序列基础模型预训练，以Mantis分词器和Transformer编码器为骨干网络。通过学生-教师框架，所提方法Utica能够学习到既捕捉增强裁剪下的时序不变性，又保留片段掩码下的细粒度局部结构的表征。该方法在UCR和UEA基准上均实现当前最优分类性能，表明非对比性方法是时间序列基础模型预训练的一种有前景且互补的策略。
原文摘要 · Abstract (English)
Self-supervised foundation models have achieved remarkable success across domains, including time series. However, the potential of non-contrastive methods, a paradigm that has driven significant advances in computer vision, remains underexplored for time series. In this work, we adapt DINOv2-style self-distillation to pretrain a time series foundation model, building on the Mantis tokenizer and transformer encoder architecture as our backbone. Through a student-teacher framework, our method Utica learns representations that capture both temporal invariance via augmented crops and fine-grained local structure via patch masking. Our approach achieves state-of-the-art classification performance on both UCR and UEA benchmarks. These results suggest that non-contrastive methods are a promising and complementary pretraining strategy for time series foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。