用纯图像训练3D模型,无需点云数据。
SimC3D: A Simple Contrastive 3D Pretraining Framework Using RGB Images
- 仅用RGB图像生成伪点云,替代昂贵的3D数据。
- 去掉2D主干网络,直接用位置嵌入提升对比学习效果。
- 在多个任务上超越依赖真实点云的模型,适合大规模预训练。
3D对比学习在点云数据上预训练后,在下游任务中表现优异。近期方法引入与3D点云相关的2D图像先验以进一步提升性能,但受限于点云数据获取成本高,可用数据集有限。为此,本文提出SimC3D,首次实现仅用纯RGB图像预训练3D主干网络。该框架具备三大优势:(1)仅使用图像数据:通过深度估计和适当的数据处理,单目合成点云展现出强大的3D预训练潜力;(2)简单框架:传统多模态框架需额外2D主干,增加计算开销。本文实证发现2D模态的核心优势在于局部信息引入,因此SimC3D直接采用2D位置嵌入作为更强的对比目标,无需2D主干,显著提升性能;(3)强性能:SimC3D在多个下游任务中优于依赖真实点云数据的先前方法,且通过融合多个图像数据集可进一步提升性能,展现出良好可扩展性。代码将公开于https://github.com/Dongjiahua/SimC3D。
原文摘要 · Abstract (English)
The 3D contrastive learning paradigm has demonstrated remarkable performance in downstream tasks through pretraining on point cloud data. Recent advances involve additional 2D image priors associated with 3D point clouds for further improvement. Nonetheless, these existing frameworks are constrained by the restricted range of available point cloud datasets, primarily due to the high costs of obtaining point cloud data. To this end, we propose SimC3D, a simple but effective 3D contrastive learning framework, for the first time, pretraining 3D backbones from pure RGB image data. SimC3D performs contrastive 3D pretraining with three appealing properties. (1) Pure image data: SimC3D simplifies the dependency of costly 3D point clouds and pretrains 3D backbones using solely RBG images. By employing depth estimation and suitable data processing, the monocular synthesized point cloud shows great potential for 3D pretraining. (2) Simple framework: Traditional multi-modal frameworks facilitate 3D pretraining with 2D priors by utilizing an additional 2D backbone, thereby increasing computational expense. In this paper, we empirically demonstrate that the primary benefit of the 2D modality stems from the incorporation of locality information. Inspired by this insightful observation, SimC3D directly employs 2D positional embeddings as a stronger contrastive objective, eliminating the necessity for 2D backbones and leading to considerable performance improvements. (3) Strong performance: SimC3D outperforms previous approaches that leverage ground-truth point cloud data for pretraining in various downstream tasks. Furthermore, the performance of SimC3D can be further enhanced by combining multiple image datasets, showcasing its significant potential for scalability. The code will be available at https://github.com/Dongjiahua/SimC3D.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。