用预对齐编码器+难样本对比学习,提升图像查3D模型的准确率。
Optimizing Multi-Modal Models for Image-Based Shape Retrieval: The Role of Pre-Alignment and Hard Contrastive Learning
- 用预对齐的图像与点云编码器,直接在共享空间检索3D形状。
- 在多个数据集上达到最优的Top1和Top10准确率,最高超基线6%。
- 无需视图合成,支持零样本和跨域检索,适合机器人与图形应用。
基于图像的3D形状检索(IBSR)旨在根据查询图像从数据库中检索3D模型,是计算机视觉、图形学和机器人领域的经典任务。现有方法通常依赖多视角渲染来弥合2D图像与3D形状间的域差距,并通过特定任务度量学习将两者嵌入共同潜空间。本文提出通过大规模多模态预训练解决IBSR问题,表明显式视图监督并非必需。受ULIP和OpenShape中预对齐图像-点云编码器启发,我们采用预对齐编码器实现零样本和标准IBSR,将图像与点云嵌入共享表示空间,通过紧凑单嵌入形状描述符进行相似性搜索。该方法跳过视图合成,天然支持零样本与跨域检索,且无需在目标数据库上重新训练。我们在零样本与监督设置下评估预对齐编码器,并引入多模态难样本对比损失(HCL)以进一步提升性能。实验表明,在多个数据集上均达到当前最优表现,尤其在OpenShape结合Point-BERT时效果最佳。此外,使用多模态HCL训练在以形状为中心的数据集上带来显著增益,凸显预训练与难样本对比学习对3D形状检索的价值。代码将通过项目网站公开。
原文摘要 · Abstract (English)
Image-based shape retrieval (IBSR) aims to retrieve 3D models from a database given a query image, hence addressing a classical task in computer vision, computer graphics, and robotics. Recent approaches typically rely on bridging the domain gap between 2D images and 3D shapes based on the use of multi-view renderings as well as task-specific metric learning to embed shapes and images into a common latent space. In contrast, we address IBSR through large-scale multi-modal pretraining and show that explicit view-based supervision is not required. Inspired by pre-aligned image--point-cloud encoders from ULIP and OpenShape that have been used for tasks such as 3D shape classification, we propose the use of pre-aligned image and shape encoders for zero-shot and standard IBSR by embedding images and point clouds into a shared representation space and performing retrieval via similarity search over compact single-embedding shape descriptors. This formulation allows skipping view synthesis and naturally enables zero-shot and cross-domain retrieval without retraining on the target database. We evaluate pre-aligned encoders in both zero-shot and supervised IBSR settings and additionally introduce a multi-modal hard contrastive loss (HCL) to further increase retrieval performance. Our evaluation demonstrates state-of-the-art performance, outperforming related methods on $Acc_{Top1}$ and $Acc_{Top10}$ for shape retrieval across multiple datasets, with best results observed for OpenShape combined with Point-BERT. Furthermore, training on our proposed multi-modal HCL yields dataset-dependent gains in standard instance retrieval tasks on shape-centric data, underscoring the value of pretraining and hard contrastive learning for 3D shape retrieval. The code will be made available via the project website.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。