从一张全景图快速生成高保真3D场景,20秒完成。
Pano3DComposer: Feed-Forward Compositional 3D Scene Generation from Single Panoramic Image

- 用可插拔模块将物体与布局解耦,直接转换3D物体世界坐标。
- 在合成与真实数据集上几何精度优于现有方法,20秒生成完整3D场景。
- 适合需要快速构建全景3D环境的设计师与开发者。
当前的组合式图像到3D场景生成方法依赖耗时的迭代布局优化或僵化的联合物体-布局生成。多数方法仅使用有限视场的透视图,难以构建完整的360度环境。为此,我们设计了Pano3DComposer,一种针对全景图的高效前馈框架。为解耦物体生成与布局估计,提出即插即用的物体-世界坐标转换预测器。该模块将现成图像到3D模型生成的3D物体从局部坐标转换至世界坐标。通过使用目标物体裁剪图、多视角物体渲染图和相机参数,基于VGGT架构改进为Alignment-VGGT进行转换预测。采用伪几何监督训练以缓解生成物与真实物体间的形状差异。对于未见领域输入,进一步引入粗到精(C2F)对齐机制,通过场景渲染反馈迭代优化几何一致性。本方法在合成与真实世界数据集上实现了图像/文本到3D任务的优越几何精度。在RTX 4090 GPU上约20秒即可生成高保真3D场景。
原文摘要 · Abstract (English)
Current compositional image-to-3D scene generation approaches construct 3D scenes by time-consuming iterative layout optimization or inflexible joint object-layout generation. Moreover, most methods rely on limited field-of-view perspective images, hindering the creation of complete 360-degree environments. To address these limitations, we design Pano3DComposer, an efficient feed-forward framework for panoramic images. To decouple object generation from layout estimation, we propose a plug-and-play Object-World Transformation Predictor. This module converts the 3D objects generated by off-the-shelf image-to-3D models from local to world coordinates. To achieve this, we adapt the VGGT architecture to Alignment-VGGT by using target object crop, multi-view object renderings and camera parameters to predict the transformation. The predictor is trained using pseudo-geometric supervision to address the shape discrepancy between generated and ground-truth objects. For input images from unseen domains, we further introduce a Coarse-to-Fine (C2F) alignment mechanism for Pano3DComposer that iteratively refines geometric consistency with feedback of scene rendering. Our method achieves superior geometric accuracy for image/text-to-3D tasks on synthetic and real-world datasets. It can generate a high-fidelity 3D scene in approximately 20 seconds on an RTX 4090 GPU. Project page: https://qiuzidian.github.io/pano3dcomposer-page/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。