从单张图片重建高保真3D场景,解决物体错位与遮挡问题。
CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image
- 分步生成:先分割物体、分析空间关系,再独立生成3D几何体。
- 通过点云与MAE条件建模,精准还原被遮挡部分的形状纹理。
- 引入物理约束图优化物体位置,避免穿透漂浮,适合机器人仿真。
从单张RGB图像恢复高质量3D场景是计算机图形学中的挑战性任务。现有方法常受限于领域特定性或生成质量低。为此,我们提出CAST(基于组件对齐的单图像3D场景重建),一种新型3D场景重建方法。CAST首先从输入图像中提取对象级2D分割和相对深度信息,随后利用基于GPT的模型分析物体间的空间关系,以理解场景中物体的相互作用,实现更连贯的重建。接着,采用遮挡感知的大规模3D生成模型,独立生成每个物体的完整几何结构,结合MAE和点云条件来缓解遮挡与部分信息的影响,确保与源图像在几何和纹理上精确对齐。为将各物体准确融入场景,对齐生成模型计算必要的变换,使生成网格能精确放置并整合到场景点云中。最后,通过细粒度关系图构建约束图,进行物理感知修正,指导物体姿态优化,确保物理一致性与空间协调性。利用有符号距离场(SDF),该方法有效解决遮挡、物体穿插及漂浮等问题,使生成场景真实反映现实物理交互。CAST可用于机器人领域,支持高效的实转虚工作流,提供真实、可扩展的仿真环境。
原文摘要 · Abstract (English)
Recovering high-quality 3D scenes from a single RGB image is a challenging task in computer graphics. Current methods often struggle with domain-specific limitations or low-quality object generation. To address these, we propose CAST (Component-Aligned 3D Scene Reconstruction from a Single RGB Image), a novel method for 3D scene reconstruction and recovery. CAST starts by extracting object-level 2D segmentation and relative depth information from the input image, followed by using a GPT-based model to analyze inter-object spatial relationships. This enables the understanding of how objects relate to each other within the scene, ensuring more coherent reconstruction. CAST then employs an occlusion-aware large-scale 3D generation model to independently generate each object's full geometry, using MAE and point cloud conditioning to mitigate the effects of occlusions and partial object information, ensuring accurate alignment with the source image's geometry and texture. To align each object with the scene, the alignment generation model computes the necessary transformations, allowing the generated meshes to be accurately placed and integrated into the scene's point cloud. Finally, CAST incorporates a physics-aware correction step that leverages a fine-grained relation graph to generate a constraint graph. This graph guides the optimization of object poses, ensuring physical consistency and spatial coherence. By utilizing Signed Distance Fields (SDF), the model effectively addresses issues such as occlusions, object penetration, and floating objects, ensuring that the generated scene accurately reflects real-world physical interactions. CAST can be leveraged in robotics, enabling efficient real-to-simulation workflows and providing realistic, scalable simulation environments for robotic systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。