单张图像同时估计物体姿态与重建3D形状,精度更高。
Glissando-Net: Deep sinGLe vIew category level poSe eStimation ANd 3D recOnstruction
- 双自编码器联合训练,实现2D图像与3D点云的交互融合。
- 在解码阶段直接预测姿态和3D形状,提升重建准确性。
- 无需代码优化,适合物体级3D重建与姿态估计任务。
我们提出一种深度学习模型Glissando-Net,可从单张RGB图像中同时估计类别级物体的姿态并重建其3D形状。以往工作多聚焦于姿态估计(常为实例级)或形状重建,但二者兼顾者少。Glissando-Net由两个自编码器组成,分别处理图像与点云数据,并联合训练。关键设计包括:1)将图像解码器生成的变换特征图注入点云编码器与解码器,增强2D-3D信息交互;2)在解码阶段同时预测3D形状与姿态,更充分利用训练阶段的点云信息。测试时丢弃3D点云编码器,仅通过图像编码器推理。该设计受codeSLAM启发,但不同于其场景重建目标,本方法专注于物体级任务,直接输出姿态不变的3D重建,无需代码优化步骤。大量实验(含消融研究与对比)验证了方法有效性,在多个基准上优于当前最优方法。
原文摘要 · Abstract (English)
We present a deep learning model, dubbed Glissando-Net, to simultaneously estimate the pose and reconstruct the 3D shape of objects at the category level from a single RGB image. Previous works predominantly focused on either estimating poses(often at the instance level), or reconstructing shapes, but not both. Glissando-Net is composed of two auto-encoders that are jointly trained, one for RGB images and the other for point clouds. We embrace two key design choices in Glissando-Net to achieve a more accurate prediction of the 3D shape and pose of the object given a single RGB image as input. First, we augment the feature maps of the point cloud encoder and decoder with transformed feature maps from the image decoder, enabling effective 2D-3D interaction in both training and prediction. Second, we predict both the 3D shape and pose of the object in the decoder stage. This way, we better utilize the information in the 3D point clouds presented only in the training stage to train the network for more accurate prediction. We jointly train the two encoder-decoders for RGB and point cloud data to learn how to pass latent features to the point cloud decoder during inference. In testing, the encoder of the 3D point cloud is discarded. The design of Glissando-Net is inspired by codeSLAM. Unlike codeSLAM, which targets 3D reconstruction of scenes, we focus on pose estimation and shape reconstruction of objects, and directly predict the object pose and a pose invariant 3D reconstruction without the need of the code optimization step. Extensive experiments, involving both ablation studies and comparison with competing methods, demonstrate the efficacy of our proposed method, and compare favorably with the state-of-the-art.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。