一个模型生成多尺寸3D表示，支持跨维度检索与零样本识别。
3D-MRL: Nested Multimodal 3D Representations via Matryoshka Representation Learning
- 用嵌套表示学习对齐点云与图文特征，实现多粒度3D表征。
- 在Objaverse-LVIS上零样本准确率提升至50.9%，优于46.8%基线。
- 单模型兼容不同算力需求，适合部署于资源差异大的场景。
视觉-语言模型将点云与图像、文本嵌入对齐，实现3D形状的零样本识别、检索和开放词汇理解。现有多模态3D预训练方法生成固定维度嵌入，需为不同计算预算分别建模。本文提出基于马特罗什卡表示学习的3D多模态预训练框架3D-MRL，通过将点云与冻结的CLIP图像和文本嵌入对齐，并在多个嵌入维度上施加对比监督，学习嵌套的3D表示。马特罗什卡目标仅作用于3D编码器，使单模型可生成不同维度的表示而无需重新训练。在Objaverse-LVIS、ModelNet40和ScanNet数据集上的实验表明，3D-MRL在零样本和少样本3D识别任务中表现优异。此外，所学表示支持单模型内跨维度检索。在Objaverse-LVIS上，Top-1准确率从46.8%提升至50.9%。检索实验进一步显示，不同嵌入维度呈现出不同程度的语义与几何特异性。
原文摘要 · Abstract (English)
Vision-Language Models align point clouds with image and text embeddings, enabling zero-shot recognition, retrieval, and open-vocabulary understanding of 3D shapes. Existing multimodal 3D pre-training methods produce fixed-dimensional embeddings, requiring separate models for different computational budgets. We propose 3D Matryoshka Representation Learning (3D-MRL), a multimodal 3D pre-training framework based on Matryoshka Representation Learning. 3D-MRL learns nested 3D representations by aligning point clouds with frozen CLIP image and text embeddings while applying contrastive supervision across multiple embedding dimensions. The Matryoshka objective is applied only to the 3D encoder, allowing a single model to produce representations at different dimensionalities without retraining. Experiments on the Objaverse-LVIS, ModelNet40, and ScanNet datasets show that 3D-MRL achieves competitive performance on zero-shot and few-shot 3D recognition tasks. In addition, the learned representations support retrieval across different embedding dimensions within a single model. On Objaverse-LVIS, 3D-MRL improves Top-1 accuracy from 46.8% to 50.9%. Retrieval experiments further show that different embedding dimensions yield varying levels of semantic and geometric specificity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。