用运动轨迹优化视频自监督学习，提升动态场景下的特征一致性。
MoSiC: Optimal-Transport Motion Trajectory for Dense Self-Supervised Learning
- 基于点追踪提取长程运动轨迹，通过最优传输机制聚类特征。
- 在6个数据集上提升1%至6%，显著改善遮挡与视角变化下的表现。
- 适合研究视频表征学习、自监督模型的开发者使用。
密集自监督学习在像素和图像块级表征学习中展现出巨大潜力，但扩展到视频仍面临运动动态复杂性的挑战。现有方法依赖静态增强，在物体形变、遮挡和相机运动下表现不佳，导致时序特征学习不一致。本文提出一种运动引导的自监督学习框架，通过聚类密集点轨迹学习时空一致的表征。利用现成点追踪器提取长程运动轨迹，并基于动量编码器的最优传输机制优化特征聚类。为确保时间一致性，沿追踪点传播聚类分配，即使在视角变化下也保持特征稳定。将运动作为隐式监督信号，使模型学习具有跨帧泛化能力的表征，在动态场景和挑战性遮挡下表现更优。通过从强图像预训练模型初始化并利用视频数据训练，我们在六个图像与视频数据集及四个评估基准上实现了1%至6%的性能提升。代码已公开于GitHub：https://github.com/SMSD75/MoSiC/tree/main。
原文摘要 · Abstract (English)
Dense self-supervised learning has shown great promise for learning pixel- and patch-level representations, but extending it to videos remains challenging due to the complexity of motion dynamics. Existing approaches struggle as they rely on static augmentations that fail under object deformations, occlusions, and camera movement, leading to inconsistent feature learning over time. We propose a motion-guided self-supervised learning framework that clusters dense point tracks to learn spatiotemporally consistent representations. By leveraging an off-the-shelf point tracker, we extract long-range motion trajectories and optimize feature clustering through a momentum-encoder-based optimal transport mechanism. To ensure temporal coherence, we propagate cluster assignments along tracked points, enforcing feature consistency across views despite viewpoint changes. Integrating motion as an implicit supervisory signal, our method learns representations that generalize across frames, improving robustness in dynamic scenes and challenging occlusion scenarios. By initializing from strong image-pretrained models and leveraging video data for training, we improve state-of-the-art by 1% to 6% on six image and video datasets and four evaluation benchmarks. The implementation is publicly available at our GitHub repository: https://github.com/SMSD75/MoSiC/tree/main
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。