用2D生成模型统一生成与感知全景图,产出可直接渲染的3D场景。
OmniX: From Unified Panoramic Generation and Perception to Graphics-Ready 3D Scenes
- 基于跨模态适配器和循环空间算子,复用2D生成先验建模全景几何、纹理与材质。
- 在大规模合成全景数据集上实现几何、光照、语义多模态无缝建模,支持真实物理渲染。
- 适合虚拟现实、游戏开发等需高保真3D场景生成的领域。
当前自动3D场景构建主要通过程序化生成和2D升维两种方式。其中,基于全景图的2D升维技术因利用强大的2D生成先验,能生成沉浸式、逼真且多样化的3D环境而备受关注。本文提出OmniX,一种面向图形级3D场景生成的统一框架,支持全景生成、感知与补全。核心思路是将2D生成模型重新用于全景图中几何、纹理及基于物理的渲染(PBR)材质的感知。不同于现有方法仅关注外观生成而忽略内在属性感知,OmniX采用跨模态适配器结构与循环空间操作符,有效复用预训练2D流匹配先验,联合建模多模态、无缝的球面矩形表示。此外,我们构建了一个大规模合成全景数据集,涵盖多种室内外场景的高质量多模态全景图。大量实验表明,OmniX在几何、光照与语义层面均具备卓越的泛化能力,实现了真正意义上的图形级3D场景生成,为沉浸式、物理真实的虚拟世界创建开辟新路径。
原文摘要 · Abstract (English)
There are two prevalent ways for automatic 3D scene construction: procedural generation and 2D lifting. Among these, panorama-based 2D lifting has emerged as a promising technique, leveraging powerful 2D generative priors to produce immersive, realistic, and diverse 3D environments. In this work, we advance this technique to generate graphics-ready 3D scenes suitable for physically based rendering (PBR), relighting, and simulation. Our key insight is to repurpose 2D generative models for panorama perception of geometry, textures, and PBR materials. Unlike existing 2D lifting approaches that emphasize appearance generation and neglect the perception of intrinsic properties, we present OmniX, a versatile and unified framework for panorama generation, perception, and completion. Built upon cross-modal adapter structure and cyclic spatial operators, OmniX effectively repurposes pre-trained 2D flow matching priors for joint modeling of multimodal, seamless equirectangular representations. Furthermore, we construct a large-scale synthetic panorama dataset comprising high-quality multimodal panoramas from diverse indoor and outdoor scenes. Extensive experiments demonstrate the effectiveness and generality of OmniX as a unified framework for panorama generation and perception across geometry, lighting, and semantics, enabling graphics-ready 3D scene generation and opening new possibilities for immersive and physically realistic virtual world creation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。