用真实火星影像生成高精度3D视频，助力太空任务模拟
Martian World Model: Controllable Video Synthesis with Physically Accurate 3D Reconstructions
- 从NASA真实火星影像重建3D地形并生成多视角视频
- 新模型在视觉质量和3D结构一致性上超越地球数据训练的模型
- 适合太空探索、机器人仿真等需要真实火星环境的场景
生成逼真的火星地貌视频对任务预演和机器人仿真至关重要。然而，由于高质量火星数据稀缺以及火星与地球影像间存在显著域差距，该任务面临独特挑战。为此，我们提出一个完整解决方案：1）多模态火星合成数据管道M3arsSynth，基于来自NASA行星数据系统（PDS）的真实立体导航图像重建3D火星环境，并渲染出高保真多视角3D视频序列；2）火星地表视频生成器MarsGen，可生成视觉逼真且几何结构与数据中编码的3D信息一致的新视频。M3arsSynth覆盖广泛火星地形与采集时间，实现米级分辨率的物理准确3D表面建模。MarsGen在该数据集上微调后，可基于初始图像帧，或可选相机轨迹/文本提示，生成新环境中的视频。实验表明，本方法优于在地球数据上训练的视频生成模型，在视觉保真度和3D结构一致性上表现更优。
原文摘要 · Abstract (English)
Synthesizing realistic Martian landscape videos is crucial for mission rehearsal and robotic simulation. However, this task poses unique challenges due to the scarcity of high-quality Martian data and the significant domain gap between Martian and terrestrial imagery. To address these challenges, we propose a holistic solution composed of two key components: 1) A data curation pipeline Multimodal Mars Synthesis (M3arsSynth), which reconstructs 3D Martian environments from real stereo navigation images, sourced from NASA's Planetary Data System (PDS), and renders high-fidelity multiview 3D video sequences. 2) A Martian terrain video generator, MarsGen, which synthesizes novel videos visually realistic and geometrically consistent with the 3D structure encoded in the data. Our M3arsSynth engine spans a wide range of Martian terrains and acquisition dates, enabling the generation of physically accurate 3D surface models at metric-scale resolution. MarsGen, fine-tuned on M3arsSynth data, synthesizes videos conditioned on an initial image frame and, optionally, camera trajectories or textual prompts, allowing for video generation in novel environments. Experimental results show that our approach outperforms video synthesis models trained on terrestrial datasets, achieving superior visual fidelity and 3D structural consistency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。