让单目视频生成3D一致的动态场景,解决画面抖动和错位问题。
Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding
- 用4D高斯点云融合2D模型先验,分阶段优化运动与语义
- 在多个任务上优于现有方法,点追踪误差降低18.7%、新视角合成质量提升
- 适合做3D动态场景理解的研究者和开发者
近期基于2D视觉的基础模型显著提升了从单目视频中分析动态场景的能力。然而,尽管具备强大泛化能力,这些模型常缺乏3D一致性,导致复杂3D环境中出现严重空间错位与时间闪烁。本文提出Motion4D框架,通过将基础模型的2D先验整合到统一的4D高斯点云表示中,解决该问题。方法包含两阶段迭代优化:1)顺序优化,分步更新运动与语义场以保持局部一致性;2)全局优化,联合精炼所有属性以实现长期连贯性。为提升运动精度,引入3D置信度图动态调整运动先验,并设计自适应重采样过程,在像素级RGB与语义误差大的区域插入新高斯点。此外,通过交替优化语义场与SAM2提示词,迭代增强语义一致性。大量实验表明,Motion4D在点追踪、视频对象分割和新视角合成等任务上显著优于2D基础模型与现有3D方法,性能全面领先。
原文摘要 · Abstract (English)
Recent advancements in foundation models for 2D vision have substantially improved the analysis of dynamic scenes from monocular videos. However, despite their strong generalization capabilities, these models often lack 3D consistency, a fundamental requirement for understanding scene geometry and motion, thereby causing severe spatial misalignment and temporal flickering in complex 3D environments. In this paper, we present Motion4D, a novel framework that addresses these challenges by integrating 2D priors from foundation models into a unified 4D Gaussian Splatting representation. Our method features a two-part iterative optimization framework: 1) Sequential optimization, which updates motion and semantic fields in consecutive stages to maintain local consistency, and 2) Global optimization, which jointly refines all attributes for long-term coherence. To enhance motion accuracy, we introduce a 3D confidence map that dynamically adjusts the motion priors, and an adaptive resampling process that inserts new Gaussians into under-represented regions based on per-pixel RGB and semantic errors. Furthermore, we enhance semantic coherence through an iterative refinement process that resolves semantic inconsistencies by alternately optimizing the semantic fields and updating prompts of SAM2. Extensive evaluations demonstrate that our Motion4D significantly outperforms both 2D foundation models and existing 3D-based approaches across diverse scene understanding tasks, including point-based tracking, video object segmentation, and novel view synthesis. Our code is available at https://hrzhou2.github.io/motion4d-web/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。