3D CoCa v2通过测试时搜索提升场景描述的泛化能力。
3D CoCa v2: Contrastive Learners with Test-Time Search for Generalizable Spatial Intelligence
- 结合对比学习与测试时搜索,不更新参数增强鲁棒性。
- 在ScanRefer和Nr3D上分别提升1.50和1.61 [email protected]。
- 适合需要跨场景泛化的3D场景理解任务。
空间智能指在三维环境中感知、推理并描述物体及其关系的能力,是具身感知与场景理解的基础。3D字幕生成旨在用自然语言描述3D场景,但因点云稀疏不规则,且现有字幕模型在不同环境间(如室内外)存在弱定位与泛化能力差的问题,仍具挑战。为此,我们提出3D CoCa v2,一种统一对比视觉-语言学习与3D字幕生成的通用框架,并通过无需更新参数的测试时搜索(TTS)进一步提升鲁棒性。3D CoCa v2基于冻结的CLIP语义先验、空间感知的3D场景编码器和联合优化对比与字幕目标的多模态解码器,避免使用外部检测器或手工提议。推理时,TTS生成多样化字幕候选,利用紧凑场景摘要进行奖励引导选择。实验显示,在ScanRefer和Nr3D上较3D CoCa分别提升+1.50和+1.61 [email protected];在TOD3Cap零样本跨分布评估中,[email protected]提升+3.8。代码将发布于https://github.com/AIGeeksGroup/3DCoCav2。
原文摘要 · Abstract (English)
Spatial intelligence refers to the ability to perceive, reason about, and describe objects and their relationships within three-dimensional environments, forming a foundation for embodied perception and scene understanding. 3D captioning aims to describe 3D scenes in natural language; however, it remains challenging due to the sparsity and irregularity of point clouds and, more critically, the weak grounding and limited out-of-distribution (OOD) generalization of existing captioners across drastically different environments, including indoor and outdoor 3D scenes. To address this challenge, we propose 3D CoCa v2, a generalizable 3D captioning framework that unifies contrastive vision-language learning with 3D caption generation and further improves robustness via test-time search (TTS) without updating the captioner parameters. 3D CoCa v2 builds on a frozen CLIP-based semantic prior, a spatially-aware 3D scene encoder for geometry, and a multimodal decoder jointly optimized with contrastive and captioning objectives, avoiding external detectors or handcrafted proposals. At inference, TTS produces diverse caption candidates and performs reward-guided selection using a compact scene summary. Experiments show improvements over 3D CoCa of +1.50 [email protected] on ScanRefer and +1.61 [email protected] on Nr3D, and +3.8 [email protected] in zero-shot OOD evaluation on TOD3Cap. Code will be released at https://github.com/AIGeeksGroup/3DCoCav2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。