用多平面同步让2D模型生成高质量360度全景图
DreamCube: 3D Panorama Generation via Multi-plane Synchronization
- 将2D模型的算子通过多平面同步拓展到全景域
- 实现多样外观与精确几何，且保持多视角一致性
- 适合做全景生成、深度估计和3D场景重建的研究者
3D全景合成是一项前景广阔但极具挑战的任务，要求生成内容具有高质量且多样的视觉外观与几何结构。现有方法利用预训练2D基础模型中的丰富图像先验来弥补3D全景数据的稀缺，但3D全景与2D单视图之间的不兼容性限制了其效果。本文证明，通过将多平面同步应用于2D基础模型的操作符，可将其能力无缝扩展至全向域。基于此设计，我们提出DreamCube——一种用于3D全景生成的多平面RGB-D扩散模型，最大限度复用2D基础模型先验，实现多样外观、准确几何并保持多视图一致性。大量实验验证了该方法在全景图像生成、全景深度估计及3D场景生成任务上的有效性。
原文摘要 · Abstract (English)
3D panorama synthesis is a promising yet challenging task that demands high-quality and diverse visual appearance and geometry of the generated omnidirectional content. Existing methods leverage rich image priors from pre-trained 2D foundation models to circumvent the scarcity of 3D panoramic data, but the incompatibility between 3D panoramas and 2D single views limits their effectiveness. In this work, we demonstrate that by applying multi-plane synchronization to the operators from 2D foundation models, their capabilities can be seamlessly extended to the omnidirectional domain. Based on this design, we further introduce DreamCube, a multi-plane RGB-D diffusion model for 3D panorama generation, which maximizes the reuse of 2D foundation model priors to achieve diverse appearances and accurate geometry while maintaining multi-view consistency. Extensive experiments demonstrate the effectiveness of our approach in panoramic image generation, panoramic depth estimation, and 3D scene generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。