单张图像一键重建场景中所有物体的3D形态与姿态,速度快且准确。
OCH3R: Object-Centric Holistic 3D Reconstruction

- 用统一的Transformer模型一次预测所有物体的类别、位置、姿态和3D形状。
- 在多个基准上达到顶尖水平,尤其在复杂场景下速度比传统方法快数十倍。
- 适合需要快速高精度3D重建的工业应用和机器人感知任务。
基于单张RGB图像的物体中心化全景3D重建是计算机视觉中的核心挑战。现有方法多采用多阶段流水线:先用预训练分割器提取单个物体,再分别进行3D重建。这类方法计算开销大,对分割错误敏感,且难以应对复杂场景。我们提出OCH3R,一个统一的物体中心化全景3D重建框架。OCH3R仅需一次前向传播,即可同时预测所有物体实例的6D姿态及其详细3D重建结果。其核心思想是基于Transformer的架构,逐像素预测包括基于CLIP的类别嵌入、度量深度、归一化物体坐标(NOCS)以及代表每个物体的固定数量3D高斯分布。为监督这些高斯重建,我们利用预测的6D姿态将它们变换到规范空间,并与预渲染的规范真值对齐,避免了每张图像生成高斯标签的高成本。在标准室内基准测试中,OCH3R在单目深度估计、开放词汇语义分割及仅使用RGB的类别级6D姿态估计任务上均达到当前最优性能,同时生成高质量、可编辑的单物体3D重建。关键优势在于推理过程完全前馈,且不受物体数量影响,因此在杂乱场景中相比传统多阶段方法实现了数量级的速度提升。
原文摘要 · Abstract (English)
Object-centric scene understanding is a fundamental challenge in computer vision. Existing approaches often rely on multi-stage pipelines that first apply pre-trained segmentors to extract individual objects, followed by per-object 3D reconstruction. Such methods are computationally expensive, fragile to segmentation errors, and scale poorly with scene complexity. We introduce OCH3R, a unified framework for Object-Centric Holistic 3D Reconstruction from a single RGB image. OCH3R performs one forward pass to simultaneously predict all object instances with their 6D poses and detailed 3D reconstructions. The key idea is a transformer architecture that predicts per-pixel attributes, including CLIP-based category embeddings, metric depth, normalized object coordinates (NOCS), and a fixed number of 3D Gaussians representing each object. To supervise these Gaussian reconstructions, we transform them into canonical space using the predicted 6D poses and align them with pre-rendered canonical ground truth, avoiding costly per-image Gaussian label generation. On standard indoor benchmarks, OCH3R achieves state-of-the-art performance across monocular depth estimation, open-vocabulary semantic segmentation, and RGB-only category-level 6D pose estimation, while producing high-fidelity, editable per-object reconstructions. Crucially, inference is fully feed-forward and scales independently of the number of objects, offering orders-of-magnitude speedups over conventional multi-stage pipelines in cluttered scenes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。