提出统一表征空间,让3D视觉定位更准
Unified Representation Space for 3D Visual Grounding
- 用CLIP模型把图像和文本映射到同一空间
- 在ScanRefer和Nr3D/Sr3D上提升至少2.24%
- 适合做3D场景理解与跨模态定位的研究者
3D视觉定位(3DVG)是场景理解的关键任务,旨在根据文本描述识别3D场景中的物体。然而,现有方法依赖独立预训练的视觉与文本编码器,导致两者在空间几何和语义类别上存在显著差异,常引发定位与分类错误。本文提出UniSpace-3D,创新性地构建3DVG的统一表征空间,有效弥合视觉与文本特征间的鸿沟。具体设计包括:i)统一表征编码器,利用预训练的CLIP模型将视觉与文本特征映射至统一空间;ii)多模态对比学习模块,进一步缩小模态差距;iii)语言引导的查询选择模块,结合位置与语义信息,精准定位与文本描述匹配的候选物体点。大量实验表明,UniSpace-3D在ScanRefer与Nr3D/Sr3D数据集上相比基线模型性能提升至少2.24%。代码将在论文接收后公开。
原文摘要 · Abstract (English)
3D visual grounding (3DVG) is a critical task in scene understanding that aims to identify objects in 3D scenes based on text descriptions. However, existing methods rely on separately pre-trained vision and text encoders, resulting in a significant gap between the two modalities in terms of spatial geometry and semantic categories. This discrepancy often causes errors in object positioning and classification. The paper proposes UniSpace-3D, which innovatively introduces a unified representation space for 3DVG, effectively bridging the gap between visual and textual features. Specifically, UniSpace-3D incorporates three innovative designs: i) a unified representation encoder that leverages the pre-trained CLIP model to map visual and textual features into a unified representation space, effectively bridging the gap between the two modalities; ii) a multi-modal contrastive learning module that further reduces the modality gap; iii) a language-guided query selection module that utilizes the positional and semantic information to identify object candidate points aligned with textual descriptions. Extensive experiments demonstrate that UniSpace-3D outperforms baseline models by at least 2.24% on the ScanRefer and Nr3D/Sr3D datasets. The code will be made available upon acceptance of the paper.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。