3D CoCa用对比学习统一生成3D场景描述，提升语义与空间理解。
3D CoCa: Contrastive Learners are 3D Captioners
- 融合对比学习与3D描述生成，共享特征空间联合优化。
- 在ScanRefer和Nr3D上分别提升10.2%和5.76%的CIDEr得分。
- 无需外部检测器或人工提议，适合3D理解与生成任务研究者。
3D场景描述旨在以自然语言描述3D场景内容，但受限于点云固有的稀疏性及现有方法中跨模态对齐弱的问题，仍具挑战。为此，我们提出3D CoCa，一种新颖的统一框架，将对比视觉-语言学习与3D描述生成无缝结合于单一架构中。该方法利用冻结的CLIP视觉-语言主干提供丰富语义先验，通过空间感知的3D场景编码器捕捉几何上下文，并采用多模态解码器生成描述性文本。不同于依赖显式物体提议的两阶段方法，3D CoCa在共享特征空间中联合优化对比与描述任务，无需外部检测器或手工提议。这种联合训练范式通过对齐3D与文本表示，增强了空间推理与语义锚定能力。在ScanRefer和Nr3D基准上的大量实验表明，3D CoCa在0.5IoU下的CIDEr指标上分别显著优于当前最优方法10.2%和5.76%。代码将发布于https://github.com/AIGeeksGroup/3DCoCa。
原文摘要 · Abstract (English)
3D captioning, which aims to describe the content of 3D scenes in natural language, remains highly challenging due to the inherent sparsity of point clouds and weak cross-modal alignment in existing methods. To address these challenges, we propose 3D CoCa, a novel unified framework that seamlessly combines contrastive vision-language learning with 3D caption generation in a single architecture. Our approach leverages a frozen CLIP vision-language backbone to provide rich semantic priors, a spatially-aware 3D scene encoder to capture geometric context, and a multi-modal decoder to generate descriptive captions. Unlike prior two-stage methods that rely on explicit object proposals, 3D CoCa jointly optimizes contrastive and captioning objectives in a shared feature space, eliminating the need for external detectors or handcrafted proposals. This joint training paradigm yields stronger spatial reasoning and richer semantic grounding by aligning 3D and textual representations. Extensive experiments on the ScanRefer and Nr3D benchmarks demonstrate that 3D CoCa significantly outperforms current state-of-the-arts by 10.2% and 5.76% in CIDEr at 0.5IoU, respectively. Code will be available at https://github.com/AIGeeksGroup/3DCoCa.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。