用视频优化3D高斯点云，实现复杂场景下动态物体的高质量4D重建。
Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild
- 通过因果潜变量保证每帧预测一致，初始化可变形3D高斯点云。
- 结合视图条件扩散先验，恢复可见细节并补全被遮挡区域。
- 适合处理大形变、严重遮挡的野外真实视频，性能显著优于现有方法。
从单目视频重建动态非刚性物体需要融合视觉线索与几何和外观的数据驱动先验。以往方法或直接从视觉输入预测4D表示，或先用先验初始化3D结构再基于视频证据变形优化。前者受限于4D训练数据稀缺，后者仅在初始阶段使用先验，后续完全依赖视频监督；两者在存在大形变和遮挡的野外场景中表现不佳。我们提出Lift4D，一种测试时优化框架，解决上述问题。首先，将现有单视图3D重建模型改造为通过因果潜变量条件生成时间一致的帧级预测，为可变形3D高斯点云提供连贯初始化。随后，通过考虑遮挡的优化过程“雕刻”该表示，精确恢复可见表面细节，并利用视图条件扩散先验完成未观测区域。实验表明，Lift4D在具有严重遮挡和非刚性运动的挑战性野外序列上明显优于现有4D重建方法。
原文摘要 · Abstract (English)
Reconstructing dynamic non-rigid objects from monocular video requires integrating visual cues from direct observations with data-driven priors over geometry and appearance. Prior approaches either learn to directly predict 4D representations from visual input or initialize a 3D representation that is subsequently deformed and refined based on video evidence. However, the former are constrained by the scarcity of 4D training data, while the latter leverage priors only for the initial reconstruction and rely solely on video supervision thereafter; neither handles complex in-the-wild scenarios with large deformations and occlusions well. We present Lift4D, a test-time optimization framework that addresses both limitations. First, we adapt an existing single-view 3D reconstruction model to yield temporally consistent per-frame predictions via causal latent conditioning, providing a coherent initialization for a deformable 3D Gaussian Splatting representation. We then ``sculpt'' this representation to match the input video through an occlusion-aware optimization that faithfully recovers visible surface details while completing unobserved regions using a view-conditioned diffusion prior. We demonstrate that Lift4D clearly improves over prior 4D reconstruction methods, particularly on challenging in-the-wild sequences with severe occlusions and non-rigid motion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。