提出PanoGaussian模型,实现单目视频下大视角变化的4D场景一致重建。
Unified Panoramic-Gaussian Representation for Monocular 4D Scene Synthesis

- 用全景轨迹引导生成,统一训练与推理框架
- 在大视角偏移下仍保持跨视图一致性
- 融合全景与动态高斯表示,捕捉物理运动先验
单目视频的4D场景合成近年取得显著进展,但现有方法受限于视图插值,难以推断未观测区域。本文将任务重新定义为包含未见区域的4D场景合成,突破传统插值范式。通过相机条件视频生成,可沿指定相机轨迹生成未见区域,但此类方法缺乏显式3D先验,且优化时采用随机相机轨迹,导致大轨迹偏移下严重不一致。为此,我们构建了基于全景轨迹引导的统一训练与推理框架,提升了跨视图一致性。然而,仅使用全景表示难以有效建模动态内容,全景空间中的物体运动引入尺度与形状畸变。为此,我们提出PanoGaussian:一种将全景表示提炼为显式动态高斯表示的统一框架,以捕捉4D场景的动态物理先验。实验表明,该方法在大视角变化下仍能实现一致的4D场景合成。
原文摘要 · Abstract (English)
4D scene synthesis from monocular videos has made significant progress in recent years. However, existing methods are typically constrained by view interpolation. As a result, they struggle to infer unseen regions beyond the observed views. In this paper, we reformulate the task as 4D scene synthesis with unseen regions, which extends beyond traditional interpolation settings. Camera-conditioned video generation enables unseen region synthesis by guiding generation along specified cameras. However, these methods lack explicit 3D priors and are optimized with random camera trajectories. This design leads to severe inconsistencies under large trajectory deviations. To address this limitation, we build a unified training and inference framework with panoramic trajectory guidance. While this design improves cross-view consistency, the panoramic representation alone fails to model dynamic content effectively. Object motion in panoramic space introduces scale and shape distortions. To address this, we propose PanoGaussian, a unified Panoramic-Gaussian representation that distills the panoramic representation into an explicit dynamic Gaussian representation to capture dynamic physical priors of the 4D scene. Experiments demonstrate that PanoGaussian achieves consistent 4D scene synthesis even under large viewpoint variations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。