arXiv:2605.26576cs.CVcs.LG2026-05

无需人工标注,实现3D高斯点云中多视角一致的指代分割。

TrackRef3D: Multi-View Consistent Track-then-Label for Open-World Referring Segmentation in 3D Gaussian Splatting

论文配图:TrackRef3D: Multi-View Consistent Track-then-Label for Open-World Referring Segmentation in 3D Gaussian Splatting
图 1 · 摘自论文原文
  • 先追踪物体轨迹再统一标注语义,解耦发现与语义定位。
  • 跨视角聚类投票生成统一语义标识,保证多视角一致性。
  • 适合开放世界场景,对不同复杂度查询均表现稳健。

指代3D高斯点云(R3DGS)利用自然语言实现3D物体分割,是具身智能的关键能力。然而现有方法依赖昂贵的逐场景人工标注和逐视角伪掩码生成,存在多视角不一致且难以泛化到不同查询粒度的问题。为此,我们提出TrackRef3D,一种完全自动化的端到端流程,通过引入多视角一致的‘先追踪后标注’范式,在无需人工标注的情况下实现3D高斯点云中的开放世界指代分割。该方法核心在于提出轨迹感知语义共识模块(TSCM),通过同义聚类与轨迹感知投票聚合跨视角预测,建立统一的语义身份以保障一致性。同时,采用可见性感知描述生成策略降低歧义,并设计混合训练策略(HTS),联合优化粗粒度类别语义与细粒度指代线索,基于多正例对比目标提升对多样化查询的鲁棒性。大量实验证明,TrackRef3D达到当前最优性能。

原文摘要 · Abstract (English)

Referring 3D Gaussian Splatting (R3DGS), which utilizes natural language for 3D object segmentation, has emerged as a crucial capability for embodied AI. However, existing methods typically rely on expensive per-scene manual annotation and per-view pseudo mask generation, which suffer from multi-view inconsistency and poor generalization to varying query specificities. To address this, we present TrackRef3D, a fully automatic pipeline that achieves open-world referring segmentation in 3D Gaussian Splatting (3DGS) without manual annotation by introducing a multi-view consistent track-then-label paradigm that fundamentally decouples object discovery from semantic grounding. Specifically, we propose a Trajectory-Aware Semantic Consensus Module (TSCM) which aggregates cross-view predictions via synonymous clustering and trajectory-aware voting to establish a canonical semantic identity, thereby ensuring multi-view consistency. Furthermore, we employ a visibility-aware description generation strategy to mitigate ambiguity and propose a Hybrid Training Strategy (HTS) that jointly optimizes coarse category semantics and fine-grained referential cues to ensure robustness under varying query specificities using a multi-positive contrastive objective. Extensive experiments on benchmarks demonstrate that TrackRef3D achieves state-of-the-art performance.

3D分割指代理解高斯溅射开放世界

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。