AnyCam通过自监督学习,从日常视频中快速恢复相机位姿与内参。
AnyCam: Learning to Recover Camera Poses and Intrinsics from Casual Videos
- 基于变压器架构的前馈模型,直接从动态视频推断相机参数。
- 在多个数据集上实现高精度位姿估计,且比现有方法快得多。
- 无需标注数据,适合处理真实世界中的复杂动态场景。
从日常视频中估计相机运动与内参是计算机视觉的核心挑战。传统基于捆绑调整的方法(如SfM和SLAM)在任意数据上表现不稳定。尽管已有针对动态场景的专用SfM方法,但它们或需已知内参,或依赖计算量大的测试时优化,性能仍有限。近期方法如Dust3r以数据驱动方式重构SfM问题,但对动态物体仍不鲁棒,且需有标签数据进行监督训练。为此,我们提出AnyCam,一种快速的变压器模型,可从前馈方式直接从动态视频序列中估计相机位姿与内参。其核心思想是让网络学习真实相机位姿的强先验。为扩大训练规模,我们采用基于不确定性的损失函数,并利用预训练的深度与光流网络,而非运动或轨迹监督,从而可使用大量未标注的YouTube视频数据。此外,通过轻量级轨迹精修步骤,确保预测轨迹不会随时间累积漂移。我们在标准数据集上验证了AnyCam,在定性和定量上均取得准确的相机位姿与内参估计。即使加入轨迹精修,AnyCam在动态场景下的SfM任务中仍显著快于现有方法。最后,结合相机信息、不确定性与深度,该模型能生成高质量4D点云。
原文摘要 · Abstract (English)
Estimating camera motion and intrinsics from casual videos is a core challenge in computer vision. Traditional bundle-adjustment based methods, such as SfM and SLAM, struggle to perform reliably on arbitrary data. Although specialized SfM approaches have been developed for handling dynamic scenes, they either require intrinsics or computationally expensive test-time optimization and often fall short in performance. Recently, methods like Dust3r have reformulated the SfM problem in a more data-driven way. While such techniques show promising results, they are still 1) not robust towards dynamic objects and 2) require labeled data for supervised training. As an alternative, we propose AnyCam, a fast transformer model that directly estimates camera poses and intrinsics from a dynamic video sequence in feed-forward fashion. Our intuition is that such a network can learn strong priors over realistic camera poses. To scale up our training, we rely on an uncertainty-based loss formulation and pre-trained depth and flow networks instead of motion or trajectory supervision. This allows us to use diverse, unlabelled video datasets obtained mostly from YouTube. Additionally, we ensure that the predicted trajectory does not accumulate drift over time through a lightweight trajectory refinement step. We test AnyCam on established datasets, where it delivers accurate camera poses and intrinsics both qualitatively and quantitatively. Furthermore, even with trajectory refinement, AnyCam is significantly faster than existing works for SfM in dynamic settings. Finally, by combining camera information, uncertainty, and depth, our model can produce high-quality 4D pointclouds.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。