用语言模型实现高效3D点云域适应,少样本下效果显著提升。
CLIPoint3D: Language-Grounded Few-Shot Unsupervised 3D Point Cloud Domain Adaptation
- 基于冻结的CLIP,通过深度图投影和轻量编码器融合语言与几何信息。
- 在两个基准上相比基线提升3%-16%准确率,且无需训练主干网络。
- 适合需要快速部署、数据稀缺的3D感知任务,如机器人抓取。
最近的视觉语言模型(如CLIP)展现出强大的跨模态推理能力,已扩展至3D感知。然而,这些模型在域偏移下仍显脆弱,尤其是在从合成到真实点云的迁移中。传统3D域适应方法依赖大量可训练编码器,虽精度高但效率低。我们提出CLIPoint3D,首个基于CLIP的少样本无监督3D点云域适应框架。该方法将3D样本投影为多张深度图,利用冻结的CLIP主干,通过融入高层语言先验与轻量3D编码器的几何线索,进行知识驱动的提示调优。为有效适应任务特征,我们对CLIP编码器采用参数高效微调,并设计熵引导的视图采样策略以选择可信投影。此外,基于最优传输的对齐损失与不确定性感知的原型对齐损失协同缩小源-目标分布差距,同时保持类别可分性。在PointDA-10和GraspNetPC-10基准上的大量实验表明,CLIPoint3D相较基于CLIP及传统编码器的基线,在性能上持续提升3%-16%。
原文摘要 · Abstract (English)
Recent vision-language models (VLMs) such as CLIP demonstrate impressive cross-modal reasoning, extending beyond images to 3D perception. Yet, these models remain fragile under domain shifts, especially when adapting from synthetic to real-world point clouds. Conventional 3D domain adaptation approaches rely on heavy trainable encoders, yielding strong accuracy but at the cost of efficiency. We introduce CLIPoint3D, the first framework for few-shot unsupervised 3D point cloud domain adaptation built upon CLIP. Our approach projects 3D samples into multiple depth maps and exploits the frozen CLIP backbone, refined through a knowledge-driven prompt tuning scheme that integrates high-level language priors with geometric cues from a lightweight 3D encoder. To adapt task-specific features effectively, we apply parameter-efficient fine-tuning to CLIP's encoders and design an entropy-guided view sampling strategy for selecting confident projections. Furthermore, an optimal transport-based alignment loss and an uncertainty-aware prototype alignment loss collaboratively bridge source-target distribution gaps while maintaining class separability. Extensive experiments on PointDA-10 and GraspNetPC-10 benchmarks show that CLIPoint3D achieves consistent 3-16% accuracy gains over both CLIP-based and conventional encoder-based baselines. Project page: https://sarthakm320.github.io/CLIPoint3D.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。