用4个摄像头实现高精度3D点跟踪，解决遮挡与深度模糊问题。
Multi-View 3D Point Tracking
- 多视角特征融合+Transformer更新，直接预测3D对应点。
- 在真实数据集上达到3.1厘米和2.0厘米的中位轨迹误差。
- 支持1-8个视角、不同拍摄角度，适合实际应用部署。
我们提出首个数据驱动的多视角3D点跟踪方法，可在动态场景中追踪任意点，仅需少量相机（如4台）即可实现鲁棒、精准的在线跟踪。相比单目追踪易受深度歧义和遮挡影响，或以往多相机方法需20台以上相机并进行逐序列优化，本方法采用前馈模型，结合已知相机位姿及传感器或估计的多视角深度，将多视图特征融合为统一点云，利用k近邻相关与Transformer更新机制，在遮挡下仍能可靠估计长距离3D对应关系。在5000个合成的Kubric多视图序列上训练，评估在两个真实世界基准数据集（Panoptic Studio 和 DexYCB）上表现优异，中位轨迹误差分别为3.1厘米和2.0厘米。方法对1-8视角、不同视角布局和24-150帧时长的视频均具良好泛化能力。项目代码与数据集已公开，旨在推动多视角3D跟踪研究新标准。
原文摘要 · Abstract (English)
We introduce the first data-driven multi-view 3D point tracker, designed to track arbitrary points in dynamic scenes using multiple camera views. Unlike existing monocular trackers, which struggle with depth ambiguities and occlusion, or prior multi-camera methods that require over 20 cameras and tedious per-sequence optimization, our feed-forward model directly predicts 3D correspondences using a practical number of cameras (e.g., four), enabling robust and accurate online tracking. Given known camera poses and either sensor-based or estimated multi-view depth, our tracker fuses multi-view features into a unified point cloud and applies k-nearest-neighbors correlation alongside a transformer-based update to reliably estimate long-range 3D correspondences, even under occlusion. We train on 5K synthetic multi-view Kubric sequences and evaluate on two real-world benchmarks: Panoptic Studio and DexYCB, achieving median trajectory errors of 3.1 cm and 2.0 cm, respectively. Our method generalizes well to diverse camera setups of 1-8 views with varying vantage points and video lengths of 24-150 frames. By releasing our tracker alongside training and evaluation datasets, we aim to set a new standard for multi-view 3D tracking research and provide a practical tool for real-world applications. Project page available at https://ethz-vlg.github.io/mvtracker.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。