用全景视频实现长时间、高保真虚拟漫游,解决传统方法视野有限问题。
OmniRoam: World Wandering via Long-Horizon Panoramic Video Generation

- 通过全景视角生成可控的长时序视频,实现全局一致的场景漫游。
- 在合成与真实数据集上均优于现有方法,视觉质量与长期一致性显著提升。
- 支持实时生成和3D重建,适合虚拟现实与数字孪生应用。
利用视频生成模型建模场景近年来受到广泛关注。然而,多数现有方法依赖透视视频模型,仅能合成场景的有限视角,导致完整性与全局一致性不足。本文提出OmniRoam,一个可控的全景视频生成框架,利用全景表示在单帧中丰富的场景覆盖以及固有的时空一致性,实现长时序场景漫游。框架包含预览阶段:由轨迹控制的视频生成模型快速生成输入图像或视频的场景概览;精修阶段:对视频进行时间扩展与空间超分辨率,生成长距离、高分辨率视频,实现高保真世界漫游。为训练模型,我们构建了两个包含合成与真实采集视频的全景视频数据集。实验表明,该框架在视觉质量、可控性与长期场景一致性方面均持续优于当前最优方法,且定性和定量结果均表现优异。此外,我们展示了框架的多种扩展,包括实时视频生成与3D重建。代码已开源。
原文摘要 · Abstract (English)
Modeling scenes using video generation models has garnered growing research interest in recent years. However, most existing approaches rely on perspective video models that synthesize only limited observations of a scene, leading to issues of completeness and global consistency. We propose OmniRoam, a controllable panoramic video generation framework that exploits the rich per-frame scene coverage and inherent long-term spatial and temporal consistency of panoramic representation, enabling long-horizon scene wandering. Our framework begins with a preview stage, where a trajectory-controlled video generation model creates a quick overview of the scene from a given input image or video. Then, in the refine stage, this video is temporally extended and spatially upsampled to produce long-range, high-resolution videos, thus enabling high-fidelity world wandering. To train our model, we introduce two panoramic video datasets that incorporate both synthetic and real-world captured videos. Experiments show that our framework consistently outperforms state-of-the-art methods in terms of visual quality, controllability, and long-term scene consistency, both qualitatively and quantitatively. We further showcase several extensions of this framework, including real-time video generation and 3D reconstruction. Code is available at https://github.com/yuhengliu02/OmniRoam.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。