arXiv:2509.24528cs.CVcs.AI2025-09被引 2

通过3D语义嵌入提升开放词汇检索,解决掩码碎片化问题。

CORE-3D: Context-aware Open-vocabulary Retrieval by Embeddings in 3D

  • 用渐进细化的SemanticSAM生成更精准的物体级掩码
  • 引入多视角上下文加权编码,显著提升语义表征能力
  • 在多个基准数据集上实现3D语义分割与语言检索性能突破

由于广泛应用,从场景中进行物体检索成为研究新趋势。现有方法通过视觉-语言模型生成2D无类别掩码并投影至3D,实现零样本、开放词汇的3D语义映射。然而,直接使用原始掩码常导致掩码碎片化和语义误分配,限制了复杂环境下的效果。为此,我们采用具有渐进粒度细化的SemanticSAM生成更准确、数量更多的物体级掩码,缓解了标准SAM等模型常见的过分割问题,提升了下游3D语义分割性能。为进一步增强语义上下文,我们提出一种基于经验权重的上下文感知CLIP编码策略,整合每个掩码的多视角信息,提供更丰富的视觉上下文。我们在多个3D场景理解任务(包括3D语义分割和语言查询物体检索)上评估该方法,在多个基准数据集上均取得显著优于现有方法的结果,验证了其有效性。

原文摘要 · Abstract (English)

Object retrieval from a scene has become a new trend of research due to its numerous applications. Recent approaches achieve zero-shot, open-vocabulary 3D semantic mapping by assigning embedding vectors to 2D class-agnostic masks generated via vision-language models (VLMs) and projecting these into 3D. However, these methods often produce fragmented masks and inaccurate semantic assignments due to the direct use of raw masks, limiting their effectiveness in complex environments. To address this, we leverage SemanticSAM with progressive granularity refinement to generate more accurate and numerous object-level masks, mitigating the over-segmentation commonly observed in mask generation models such as vanilla SAM, and improving downstream 3D semantic segmentation. To further enhance semantic context, we employ a context-aware CLIP encoding strategy that integrates multiple contextual views of each mask using empirically determined weighting, providing much richer visual context. We evaluate our approach on multiple 3D scene understanding tasks, including 3D semantic segmentation and object retrieval from language queries, across several benchmark datasets. Experimental results demonstrate significant improvements over existing methods, highlighting the effectiveness of our approach.

3D语义开放词汇掩码优化CLIP

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。