同时实现世界坐标系下的4D重建与跟踪，无需依赖大量标注数据。
St4RTrack: Simultaneous 4D Reconstruction and Tracking in the World
- 通过预测两帧的点图，统一建模静态与动态场景几何。
- 在无4D真值监督下，用重投影损失实现长程对应关系计算。
- 适用于需要高精度3D动态追踪的自动驾驶、机器人等场景。
动态3D重建与点跟踪通常被视为独立任务，尽管二者密切相关。本文提出St4RTrack，一种前馈框架，从RGB视频输入中同步实现世界坐标系下的4D重建与跟踪。方法通过预测一对不同时刻帧的点图（pointmaps）来实现，且两个点图均定义在同一时刻、同一世界坐标系中，同时捕捉静态与动态场景结构，并保持三维对应关系。将这些预测沿视频序列以参考帧为基准串联，自然获得长程对应关系，从而融合3D重建与3D跟踪。不同于依赖4D真值监督的已有方法，本工作采用基于重投影损失的新颖自适应方案。我们建立了一个新的大规模世界坐标系重建与跟踪基准，验证了该统一、数据驱动框架的有效性与高效性。代码、模型与基准将公开发布。
原文摘要 · Abstract (English)
Dynamic 3D reconstruction and point tracking in videos are typically treated as separate tasks, despite their deep connection. We propose St4RTrack, a feed-forward framework that simultaneously reconstructs and tracks dynamic video content in a world coordinate frame from RGB inputs. This is achieved by predicting two appropriately defined pointmaps for a pair of frames captured at different moments. Specifically, we predict both pointmaps at the same moment, in the same world, capturing both static and dynamic scene geometry while maintaining 3D correspondences. Chaining these predictions through the video sequence with respect to a reference frame naturally computes long-range correspondences, effectively combining 3D reconstruction with 3D tracking. Unlike prior methods that rely heavily on 4D ground truth supervision, we employ a novel adaptation scheme based on a reprojection loss. We establish a new extensive benchmark for world-frame reconstruction and tracking, demonstrating the effectiveness and efficiency of our unified, data-driven framework. Our code, model, and benchmark will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。