用轨迹信息显式建模运动，提升视频表征学习效果
TrackMAE: Video Representation Learning via Track Mask and Predict
- 用点追踪生成运动轨迹，作为重建信号增强表示学习
- 在6个数据集上优于现有自监督方法，尤其擅长运动相关任务
- 适合需要精细运动感知的下游应用，如动作识别与跟踪
掩码视频建模（MVM）作为一种简单且可扩展的自监督预训练范式，仅隐式编码运动信息，限制了对时间动态的表征能力，导致在依赖细粒度运动感知的任务上表现不佳。为此，我们提出TrackMAE，一种显式利用运动信息作为重建信号的掩码视频建模方法。TrackMAE通过现成的点追踪器稀疏追踪输入视频中的关键点，生成运动轨迹，并基于这些轨迹设计运动感知的掩码策略，改进随机管状掩码。在像素和特征语义重构空间中，通过提供互补的运动目标监督信号，增强视频表征。在六个不同下游任务的数据集上评估，TrackMAE始终优于当前最先进的自监督视频学习基线，学习到更具判别性和泛化能力的表示。代码已公开于https://github.com/rvandeghen/TrackMAE。
原文摘要 · Abstract (English)
Masked video modeling (MVM) has emerged as a simple and scalable self-supervised pretraining paradigm, but only encodes motion information implicitly, limiting the encoding of temporal dynamics in the learned representations. As a result, such models struggle on motion-centric tasks that require fine-grained motion awareness. To address this, we propose TrackMAE, a simple masked video modeling paradigm that explicitly uses motion information as a reconstruction signal. In TrackMAE, we use an off-the-shelf point tracker to sparsely track points in the input videos, generating motion trajectories. Furthermore, we exploit the extracted trajectories to improve random tube masking with a motion-aware masking strategy. We enhance video representations learned in both pixel and feature semantic reconstruction spaces by providing a complementary supervision signal in the form of motion targets. We evaluate on six datasets across diverse downstream settings and find that TrackMAE consistently outperforms state-of-the-art video self-supervised learning baselines, learning more discriminative and generalizable representations. Code available at https://github.com/rvandeghen/TrackMAE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。