用3D对齐对应关系实现精准点跟踪，效果超越现有方法。
PointSt3R: Point Tracking through 3D Grounded Correspondence
- 基于3D重建模型，结合可见性头与重建损失训练点跟踪。
- 在四大数据集上表现优异，如EgoPoints提升7.1%、RGB-S提升4.2%。
- 仅用少量合成数据微调，适合静态与动态点混合场景。
近期基于3D重建的模型（如DUSt3R和MASt3R）在静态场景的2D/3D对应关系上表现突出。本文提出将此类模型用于点跟踪任务，聚焦于静态点时性能媲美甚至超越现有方法（EgoPoints上提升33.5%）。通过结合重建损失与动态对应训练，并引入可见性头，仅用少量合成数据微调MASt3R即可实现高效点跟踪。训练与评估仅在包含查询点的帧对间进行，不依赖时间上下文。利用静态与动态点混合对应，模型在四个数据集上达到竞争性或更优结果：例如在TAP-Vid-DAVIS上，δ_avg达73.8，遮挡准确率85.8%，优于CoTracker2（75.7 / 88.3%）；在EgoPoints上61.3比54.2高，在RGB-S上87.0比82.8高。还提供了3D点跟踪结果及训练数据与动态对应比例的消融分析。
原文摘要 · Abstract (English)
Recent advances in foundational 3D reconstruction models, such as DUSt3R and MASt3R, have shown great potential in 2D and 3D correspondence in static scenes. In this paper, we propose to adapt them for the task of point tracking through 3D grounded correspondence. We first demonstrate that these models are competitive point trackers when focusing on static points, present in current point tracking benchmarks ($+33.5\%$ on EgoPoints vs. CoTracker2). We propose to combine the reconstruction loss with training for dynamic correspondence along with a visibility head, and fine-tuning MASt3R for point tracking using a relatively small amount of synthetic data. Importantly, we only train and evaluate on pairs of frames where one contains the query point, effectively removing any temporal context. Using a mix of dynamic and static point correspondences, we achieve competitive or superior point tracking results on four datasets (e.g. competitive on TAP-Vid-DAVIS 73.8 $δ_{avg}$ / 85.8\% occlusion acc. for PointSt3R compared to 75.7 / 88.3\% for CoTracker2; and significantly outperform CoTracker3 on EgoPoints 61.3 vs 54.2 and RGB-S 87.0 vs 82.8). We also present results on 3D point tracking along with several ablations on training datasets and percentage of dynamic correspondences.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。