单图生成3D场景,兼顾几何精确与纹理真实
Towards Geometric and Textural Consistency 3D Scene Generation via Single Image-guided Model Generation and Layout Optimization
- 三阶段流程:先修复图像,再推导空间结构,最后优化布局
- 在多个物体场景中,几何精度和纹理保真度均优于现有方法
- 适合需要高保真3D场景生成的研究者与工业应用
近年来,3D生成在学术界和工业界取得显著进展。然而,从单张RGB图像生成3D场景仍面临挑战,现有方法在多物体场景中难以同时保证物体生成质量与场景一致性。为此,我们提出一种新颖的三阶段框架,通过单图引导的模型生成与布局优化,实现具有显式几何表示和高质量纹理细节的3D场景生成。方法首先进行图像实例分割与修补,恢复输入图像中被遮挡物体的缺失细节,从而完整生成前景3D资产。随后,通过构建伪立体视点估计相机参数与场景深度,结合模型选择策略,确保前步生成的3D资产与输入图像最优对齐。最后,通过模型参数化及点云在3D与2D空间间切比雪夫距离最小化,优化布局参数,生成与输入图像精准对齐的显式3D场景表示。在多物体场景图像集上的大量实验表明,本方法不仅在个体3D模型的几何准确性和纹理保真度上超越当前最优方法,还在场景布局合成方面具有显著优势。
原文摘要 · Abstract (English)
In recent years, 3D generation has made great strides in both academia and industry. However, generating 3D scenes from a single RGB image remains a significant challenge, as current approaches often struggle to ensure both object generation quality and scene coherence in multi-object scenarios. To overcome these limitations, we propose a novel three-stage framework for 3D scene generation with explicit geometric representations and high-quality textural details via single image-guided model generation and spatial layout optimization. Our method begins with an image instance segmentation and inpainting phase, which recovers missing details of occluded objects in the input images, thereby achieving complete generation of foreground 3D assets. Subsequently, our approach captures the spatial geometry of reference image by constructing pseudo-stereo viewpoint for camera parameter estimation and scene depth inference, while employing a model selection strategy to ensure optimal alignment between the 3D assets generated in the previous step and the input. Finally, through model parameterization and minimization of the Chamfer distance between point clouds in 3D and 2D space, our approach optimizes layout parameters to produce an explicit 3D scene representation that maintains precise alignment with input guidance image. Extensive experiments on multi-object scene image sets have demonstrated that our approach not only outperforms state-of-the-art methods in terms of geometric accuracy and texture fidelity of individual generated 3D models, but also has significant advantages in scene layout synthesis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。