用自监督点云编码器替代昂贵多模态模型，显著降低3D大模型训练成本
On the Efficacy of Self-Supervised Point Cloud Encoders for Efficient 3D Large Language Models
- 采用自监督点云编码器（如PCP-MAE）配合冻结/微调策略训练3D-LLM
- 随机初始化编码器经端到端训练后达到52.5%开放词汇准确率，接近预训练效果
- 编码器架构与预训练目标存在强交互作用，需针对性匹配
3D点云语言模型通过结合点云编码器与大语言模型实现三维理解，但现有方法依赖高成本多模态编码器（如ULIP-2），需在8x A100规模算力上进行图像-文本-点云对齐，研究与部署门槛高。本文系统考察低成本自监督点云编码器（PCP-MAE、Point-MAE）是否可作为有效替代。以MiniGPT-3D为测试平台，评估7种初始化/预训练设置（1个多模态基线、5个自监督、1个随机初始化），涵盖冻结与非冻结微调（共12组），涉及2种架构（MaskTransformer、PointTransformer）、3种预训练目标（PCP-MAE、Point-MAE、随机初始化），以及2个数据集（Objaverse 660K、ShapeNet55-34约5万）。实验揭示三大发现：(1) 四阶段MiniGPT-3D流程可从随机初始化有效训练3D编码器：端到端训练的随机初始化编码器达52.50%开放词汇准确率与44.45分句生成得分，接近顶级预训练模型；(2) 架构与预训练目标间存在显著交叉作用：PCP-MAE+MaskTransformer达59.00%准确率（最佳自监督），而Point-MAE+MaskTransformer降至46.50%，该模式在PointTransformer上反转；(3) 纯几何编码器在封闭集ModelNet40分类任务中仍表现薄弱，即使端到端微调后准确率仅约13-18%，远低于多模态基线的~62%。结果为低成本3D-LLM设计提供实用指导，并揭示自监督目标与架构间的交互规律。
原文摘要 · Abstract (English)
3D point cloud-language models (3D-LLMs) enable 3D understanding by pairing point cloud encoders with large language models, but existing methods rely on costly multi-modal encoders (e.g., ULIP-2) that require image-text-point cloud alignment on 8x A100-scale compute, creating high barriers for research and deployment. In this work, we systematically investigate whether low-cost self-supervised point cloud encoders, specifically PCP-MAE and Point-MAE, can serve as effective alternatives. Using MiniGPT-3D as our testbed, we evaluate 7 encoder initialization/pre-training setups (1 multi-modal baseline, 5 self-supervised, 1 random init) under frozen and unfrozen fine-tuning (12 total groups), across 2 architectures (MaskTransformer, PointTransformer), 3 objectives (PCP-MAE, Point-MAE, random init), and 2 datasets (Objaverse 660K, ShapeNet55-34 approximately 50K). Our experiments reveal three key findings: (1) The four-stage MiniGPT-3D pipeline can effectively train a 3D encoder from random initialization: an end-to-end trained random init encoder reaches 52.50% open-vocabulary accuracy and 44.45 captioning score, approaching top pre-trained variants; (2) Architecture and pre-training objective show strong crossover interaction: PCP-MAE + MaskTransformer achieves 59.00% accuracy (best self-supervised), while Point-MAE + MaskTransformer drops to 46.50%, with the pattern reversed for PointTransformer; (3) Closed-set ModelNet40 classification remains a core weakness of purely geometric encoders, reaching only ~13-18% accuracy vs. ~62% for the multi-modal baseline, even after end-to-end fine-tuning. Our results offer practical guidelines for cost-effective 3D-LLM design and reveal interaction patterns between self-supervised objectives and encoder architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。