用3D高斯点追踪视频，零样本即达顶尖跟踪效果
Tracking by Predicting 3-D Gaussians Over Time
- 将视频建模为随时间运动的3D高斯点集，引入合理先验
- 预训练后直接实现零样本跟踪，性能媲美最先进方法
- 微调后在Kinetics和Kubric上分别提升34.6%和13.1%
我们提出视频高斯掩码自编码器（Video-GMAE），一种自监督表示学习方法，将图像序列编码为随时间移动的高斯点集。将视频表示为高斯点集，施加了合理的归纳偏置：二维视频通常是动态三维场景的一致投影。我们发现，在此架构上预训练网络时，跟踪能力自然涌现。将学习到的高斯点轨迹映射到图像平面，即可获得与现有最优方法相当的零样本跟踪性能。经过小规模微调，模型在Kinetics数据集上提升34.6%，在Kubric数据集上提升13.1%，超越现有自监督视频方法。项目页面和代码已公开于https://videogmae.org/ 和 https://github.com/tekotan/video-gmae。
原文摘要 · Abstract (English)
We propose Video Gaussian Masked Autoencoders (Video-GMAE), a self-supervised approach for representation learning that encodes a sequence of images into a set of Gaussian splats moving over time. Representing a video as a set of Gaussians enforces a reasonable inductive bias: that 2-D videos are often consistent projections of a dynamic 3-D scene. We find that tracking emerges when pretraining a network with this architecture. Mapping the trajectory of the learnt Gaussians onto the image plane gives zero-shot tracking performance comparable to state-of-the-art. With small-scale finetuning, our models achieve 34.6% improvement on Kinetics, and 13.1% on Kubric datasets, surpassing existing self-supervised video approaches. The project page and code are publicly available at https://videogmae.org/ and https://github.com/tekotan/video-gmae.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。