通过随机丢弃时间序列片段，提升模型预训练效率与泛化能力
Enhancing Masked Time-Series Modeling via Dropping Patches
- 随机丢弃时间序列的子段，增强模型对不完整数据的适应性
- 预训练效率提升一个数量级，跨域和少样本场景表现更优
- 适合需要高效训练和鲁棒性的时序建模任务
本文探讨如何通过随机丢弃时间序列的子序列级片段来增强现有的掩码时间序列建模方法。基于此，提出一种简单而有效的方法DropPatch，具有两大优势：1）预训练效率提升一个数量级；2）在同域、跨域、少样本学习及冷启动等场景下均表现出额外优势。通过全面实验验证了该方法的有效性并分析其内在机制。实证表明，DropPatch增强了注意力机制，降低了信息冗余，并作为高效的的数据增强手段。理论上证明，随机丢弃片段可减缓Transformer表示向秩-1线性子空间坍缩的速度，从而优化学习表示的质量。
原文摘要 · Abstract (English)
This paper explores how to enhance existing masked time-series modeling by randomly dropping sub-sequence level patches of time series. On this basis, a simple yet effective method named DropPatch is proposed, which has two remarkable advantages: 1) It improves the pre-training efficiency by a square-level advantage; 2) It provides additional advantages for modeling in scenarios such as in-domain, cross-domain, few-shot learning and cold start. This paper conducts comprehensive experiments to verify the effectiveness of the method and analyze its internal mechanism. Empirically, DropPatch strengthens the attention mechanism, reduces information redundancy and serves as an efficient means of data augmentation. Theoretically, it is proved that DropPatch slows down the rate at which the Transformer representations collapse into the rank-1 linear subspace by randomly dropping patches, thus optimizing the quality of the learned representations
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。