用4D变分自编码器联合重建视频的3D结构与稠密运动，效果领先。
MotionCrafter: Dense Geometry and Motion Reconstruction with a 4D VAE
- 设计4D VAE统一建模3D点云与场景流，共享坐标系
- 几何与运动重建分别提升38.64%和25.0%，无需后处理
- 摆脱对齐限制，新归一化策略更好利用扩散先验
我们提出MotionCrafter，一个利用视频生成模型联合重建单目视频中4D几何结构与稠密运动的框架。核心思想是在统一坐标系下联合表示密集3D点图与3D场景流，并设计适配该表示的4D变分自编码器（VAE）。不同于以往严格对齐3D潜空间与RGB VAE潜空间（尽管分布本质不同），我们证明这种对齐非必要且可能损害性能。为此，提出新的数据归一化与训练策略，更有效地传递扩散模型先验，显著提升重建质量。在多个数据集上的大量实验表明，MotionCrafter在几何重建和稠密场景流估计上均达当前最优水平，分别提升38.64%和25.0%，且无需任何后优化。
原文摘要 · Abstract (English)
We present MotionCrafter, a framework that leverages video generators to jointly reconstruct 4D geometry and estimate dense motion from a monocular video. The key idea is a joint representation of dense 3D point maps and 3D scene flows in a shared coordinate system, together with a 4D VAE tailored to learn this representation effectively. Unlike prior work that strictly aligns 3D values and latents with RGB VAE latents-despite their fundamentally different distributions-we show that such alignment is unnecessary and can hurt performance. Instead, we propose a new data normalization and VAE training strategy that better transfers diffusion priors and greatly improves reconstruction quality. Extensive experiments on multiple datasets show that MotionCrafter achieves state-of-the-art performance in both geometry reconstruction and dense scene flow estimation, delivering 38.64% and 25.0% improvements in geometry and motion reconstruction, respectively, all without any post-optimization. Project page: https://ruijiezhu94.github.io/MotionCrafter_Page
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。