让3D场景理解更懂上下文,解决多视角分割不一致问题
CAGS: Open-Vocabulary 3D Scene Understanding with Context-Aware Gaussian Splatting
- 构建局部图传播上下文特征,统一多视角对象分割粒度
- 在LERF-OVS和ScanNet上实例分割精度提升,碎片化错误减少
- 适合需要语言驱动3D理解的机器人与AR应用
开放词汇3D场景理解对机器人、增强现实等自然语言驱动的空间解析应用至关重要。尽管3D高斯点阵(3DGS)提供了强大的场景重建表示,但与开放词汇框架结合时面临一个关键挑战:跨视角粒度不一致。这一问题源于2D分割方法(如SAM),导致同一物体在不同视角中被分割为不同粒度(如‘咖啡套装’在一处被视作整体,另一处却拆分为‘杯+咖啡+勺’)。现有基于3DGS的方法通常依赖孤立的单高斯特征学习,忽视了连贯对象推理所需的时空上下文,造成表征碎片化。本文提出上下文感知高斯点阵(CAGS),通过构建局部图在高斯间传播上下文特征,降低不一致粒度带来的噪声;采用以掩码为中心的对比学习,在多视角间平滑SAM生成的特征;并引入预计算策略,预先构建邻近关系,显著降低大场景训练的计算开销。通过整合空间上下文,CAGS在LERF-OVS和ScanNet数据集上显著提升3D实例分割性能,减少碎片化误差,实现稳健的语言引导3D场景理解。
原文摘要 · Abstract (English)
Open-vocabulary 3D scene understanding is crucial for applications requiring natural language-driven spatial interpretation, such as robotics and augmented reality. While 3D Gaussian Splatting (3DGS) offers a powerful representation for scene reconstruction, integrating it with open-vocabulary frameworks reveals a key challenge: cross-view granularity inconsistency. This issue, stemming from 2D segmentation methods like SAM, results in inconsistent object segmentations across views (e.g., a "coffee set" segmented as a single entity in one view but as "cup + coffee + spoon" in another). Existing 3DGS-based methods often rely on isolated per-Gaussian feature learning, neglecting the spatial context needed for cohesive object reasoning, leading to fragmented representations. We propose Context-Aware Gaussian Splatting (CAGS), a novel framework that addresses this challenge by incorporating spatial context into 3DGS. CAGS constructs local graphs to propagate contextual features across Gaussians, reducing noise from inconsistent granularity, employs mask-centric contrastive learning to smooth SAM-derived features across views, and leverages a precomputation strategy to reduce computational cost by precomputing neighborhood relationships, enabling efficient training in large-scale scenes. By integrating spatial context, CAGS significantly improves 3D instance segmentation and reduces fragmentation errors on datasets like LERF-OVS and ScanNet, enabling robust language-guided 3D scene understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。