通过时间对应关系提升视频自监督学习效果
When the Future Becomes the Past: Taming Temporal Correspondence for Self-supervised Video Representation Learning
- 采用夹心式采样减少重建不确定性
- 在潜在空间恢复表示，增强语义信息
- 适合视频表征学习与下游任务应用
过去十年，视频自监督学习取得显著进展。近期方法普遍采用掩码视频建模（MVM）范式，但在无标注情况下，随机时间采样引入不确定性，增加训练难度；且现有MVM方法多在像素空间恢复掩码块，信息压缩不足。为此，我们提出基于时间对应关系的视频表征学习框架T-CoRe。针对第一挑战，设计夹心采样策略，通过选取两个辅助帧实现双向压缩，降低重建不确定性；针对第二挑战，在自蒸馏架构中引入辅助分支，在潜在空间恢复表示，生成富含时间信息的高层语义表征。T-CoRe在多个下游任务上持续表现优异，验证了其有效性。代码已开源。
原文摘要 · Abstract (English)
The past decade has witnessed notable achievements in self-supervised learning for video tasks. Recent efforts typically adopt the Masked Video Modeling (MVM) paradigm, leading to significant progress on multiple video tasks. However, two critical challenges remain: 1) Without human annotations, the random temporal sampling introduces uncertainty, increasing the difficulty of model training. 2) Previous MVM methods primarily recover the masked patches in the pixel space, leading to insufficient information compression for downstream tasks. To address these challenges jointly, we propose a self-supervised framework that leverages Temporal Correspondence for video Representation learning (T-CoRe). For challenge 1), we propose a sandwich sampling strategy that selects two auxiliary frames to reduce reconstruction uncertainty in a two-side-squeezing manner. Addressing challenge 2), we introduce an auxiliary branch into a self-distillation architecture to restore representations in the latent space, generating high-level semantic representations enriched with temporal information. Experiments of T-CoRe consistently present superior performance across several downstream tasks, demonstrating its effectiveness for video representation learning. The code is available at https://github.com/yafeng19/T-CORE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。