仅用一张图生成沉浸式3D世界，无需大量训练。
A Recipe for Generating 3D Worlds From a Single Image
- 用预训练扩散模型生成连贯全景图，再通过深度估计转为3D。
- 结合渲染点云条件，让修补模型填补未观察区域，效果逼真。
- 适合想快速构建3D场景的VR开发者，尤其擅长真实图像生成。
我们提出一种从单张图像生成沉浸式3D世界的方法，将任务视为2D修复模型的上下文学习问题。该方法几乎无需训练，直接利用现有生成模型。流程包含两步：首先使用预训练扩散模型生成连贯全景图，再通过度量深度估计将其提升至3D；随后通过将修复模型条件于渲染点云，填补未观测区域，仅需少量微调。在合成与真实图像上测试均表现优异，生成的3D环境质量高，适合用于虚拟现实展示。通过从一开始就显式建模生成环境的3D结构，本方法在多个定量图像质量指标上持续优于当前最先进的基于视频合成的方法。
原文摘要 · Abstract (English)
We introduce a recipe for generating immersive 3D worlds from a single image by framing the task as an in-context learning problem for 2D inpainting models. This approach requires minimal training and uses existing generative models. Our process involves two steps: generating coherent panoramas using a pre-trained diffusion model and lifting these into 3D with a metric depth estimator. We then fill unobserved regions by conditioning the inpainting model on rendered point clouds, requiring minimal fine-tuning. Tested on both synthetic and real images, our method produces high-quality 3D environments suitable for VR display. By explicitly modeling the 3D structure of the generated environment from the start, our approach consistently outperforms state-of-the-art, video synthesis-based methods along multiple quantitative image quality metrics. Project Page: https://katjaschwarz.github.io/worlds/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。