用图像语义增强点云，让3D语言定位更准
Hierarchical Collaborative Fusion for 3D Instance-aware Referring Expression Segmentation
- 用SAM和CLIP在像素和实例层面提取视觉语义
- 在ScanRefer和Multi3DRefer上达到新最好效果
- 适合做3D场景中精细语言定位的研究者
通用3D指代表达分割（3D-GRES）根据自然语言描述定位3D场景中的物体，即使描述匹配多个或零个目标。现有方法仅依赖稀疏点云，缺乏丰富的视觉语义信息，难以处理细粒度描述。本文提出HCF-RES，一个双创新的多模态框架：首先，层次化视觉语义分解利用SAM实例掩码，引导CLIP在像素级和实例级两个粒度上编码，保留2D到3D投影过程中的物体边界；其次，渐进式多层次融合通过模态内协作、2D语义与3D几何特征间的跨模态自适应加权，以及语言引导的精修，实现特征整合。HCF-RES在ScanRefer和Multi3DRefer数据集上均达到当前最优性能。
原文摘要 · Abstract (English)
Generalised 3D Referring Expression Segmentation (3D-GRES) localizes objects in 3D scenes based on natural language, even when descriptions match multiple or zero targets. Existing methods rely solely on sparse point clouds, lacking rich visual semantics for fine-grained descriptions. We propose HCF-RES, a multi-modal framework with two key innovations. First, Hierarchical Visual Semantic Decomposition leverages SAM instance masks to guide CLIP encoding at dual granularities -- pixel-level and instance-level features -- preserving object boundaries during 2D-to-3D projection. Second, Progressive Multi-level Fusion integrates representations through intra-modal collaboration, cross-modal adaptive weighting between 2D semantic and 3D geometric features, and language-guided refinement. HCF-RES achieves state-of-the-art results on both ScanRefer and Multi3DRefer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。