从单张图生成动态4D场景,兼顾几何与运动一致性。
Geometry-Aware Single-Image 4D Synthesis via Dense Trajectory Generation
- 用密集4D轨迹建模场景,通过扩散过程联合生成几何与运动。
- 在6万条视频数据上训练,生成结果时空连贯性显著提升。
- 适合做4D内容生成、虚拟拍摄与数字孪生的科研与工程人员。
从单张静态图像生成可交互的动态4D场景仍是核心挑战。现有生成-重建或重建-生成方法将几何与运动分离,导致时空不一致且泛化能力差。为此,我们提出MoGe4D(Motion and Geometry-Aware image-to-4D Synthesis),一种基于几何条件的单图4D生成框架,将场景建模为密集4D点轨迹。不同于分阶段处理,本方法从输入图像推断初始几何先验,并在扩散过程中预测随时间变化的轨迹,增强时空一致性并保持结构稳定。为此,我们首先构建了包含60,000个视频样本的大型数据集TrajScene-60K,提供密集4D点轨迹标注,缓解高质量4D生成训练数据稀缺问题。在此基础上,提出的4D-STraG(4D Scene Trajectory Generator)利用扩散模型生成几何一致、运动合理的目标轨迹场,引入深度引导的运动归一化策略降低尺度模糊性,并设计运动感知模块(MPM)注入运动先验。进一步提出4D-ViSM模块,可将生成的4D表示渲染为任意相机轨迹下的视频。实验表明,MoGe4D能从单图生成高质量4D场景,具备强时序连贯性、几何感知一致性及出色的零样本视图合成能力。代码已开源:https://github.com/Zhangyr2022/MoGe4D。
原文摘要 · Abstract (English)
Generating interactive and dynamic 4D scenes from a single static image remains a core challenge. Most existing generate-then-reconstruct and reconstruct-then-generate methods decouple geometry from motion, causing spatiotemporal inconsistencies and poor generalization. To address these, we present MoGe4D (Motion and Geometry-Aware image-to-4D Synthesis), a geometry-conditioned framework for single-image 4D synthesis that models a scene as dense 4D point trajectories. Instead of treating geometry and dynamics as two disconnected stages, our method starts from an initial geometric prior inferred from the input image and predicts future time-varying trajectories in a diffusion process, improving spatiotemporal coherence while preserving structural stability. To support this task, we first introduce TrajScene-60K, a large-scale dataset of 60,000 video samples with dense 4D point trajectories, addressing the scarcity of high-quality training data for scene-level 4D generation. Built on this, our diffusion-based 4D Scene Trajectory Generator (4D-STraG) predicts geometry-consistent and motion-plausible trajectory fields conditioned on the input image, with a depth-guided motion normalization strategy to reduce scale ambiguity and a Motion Perception Module (MPM) to inject motion-aware priors. We further propose a 4D View Synthesis Module (4D-ViSM) to render the generated 4D representation into videos under arbitrary camera trajectories. Experiments show that MoGe4D produces high-quality 4D scenes with strong temporal coherence, favorable geometry-aware consistency, and compelling novel-view synthesis from a single image. Code: https://github.com/Zhangyr2022/MoGe4D.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。