用2D CLIP统一处理图像、文本和点云,实现高效3D视觉定位。
TriCLIP-3D: A Unified Parameter-Efficient Framework for Tri-Modal 3D Visual Grounding based on CLIP
- 基于CLIP适配器微调,统一处理三模态数据
- 参数量减少58%,3D检测提升6.52%
- 适合追求轻量化与多模态融合的机器人研究者
3D视觉定位使智能体能理解真实3D环境中的视觉信息,对具身智能至关重要。现有方法通常使用独立编码器处理不同模态(如RGB图像、文本、3D点云),导致模型庞大且训练低效。尽管部分方法尝试使用预训练的2D多模态模型(如CLIP)处理3D任务,但点云与2D编码器的对齐仍困难,仍需依赖3D编码器提取特征,进一步增加复杂性。本文提出一种统一的2D预训练多模态框架,通过适配器微调的2D CLIP模型处理三种模态。设计几何感知的2D-3D特征恢复与融合模块(GARF),有效融合点云与图像的多尺度几何特征,并整合文本特征进行最终融合,结合多模态解码器实现深层跨模态理解。该方法实现了三模态端到端统一特征提取与融合,在保持高效的同时,相较基线减少约58%可训练参数,3D检测任务提升6.52%,3D视觉定位任务提升6.25%。
原文摘要 · Abstract (English)
3D visual grounding allows an embodied agent to understand visual information in real-world 3D environments based on human instructions, which is crucial for embodied intelligence. Existing 3D visual grounding methods typically rely on separate encoders for different modalities (e.g., RGB images, text, and 3D point clouds), resulting in large and complex models that are inefficient to train. While some approaches use pre-trained 2D multi-modal models like CLIP for 3D tasks, they still struggle with aligning point cloud data to 2D encoders. As a result, these methods continue to depend on 3D encoders for feature extraction, further increasing model complexity and training inefficiency. In this paper, we propose a unified 2D pre-trained multi-modal network to process all three modalities (RGB images, text, and point clouds), significantly simplifying the architecture. By leveraging a 2D CLIP bi-modal model with adapter-based fine-tuning, this framework effectively adapts to the tri-modal setting, improving both adaptability and performance across modalities. Our Geometric-Aware 2D-3D Feature Recovery and Fusion (GARF) module is designed to fuse geometric multi-scale features from point clouds and images. We then integrate textual features for final modality fusion and introduce a multi-modal decoder to facilitate deep cross-modal understanding. Together, our method achieves unified feature extraction and fusion across the three modalities, enabling an end-to-end 3D visual grounding model. Compared to the baseline, our method reduces the number of trainable parameters by approximately 58\%, while achieving a 6.52\% improvement in the 3D detection task and a 6.25\% improvement in the 3D visual grounding task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。