用投票机制提升3D高斯点云的场景理解能力
VoteSplat: Hough Voting Gaussian Splatting for 3D Scene Understanding
- 结合霍夫投票与3D高斯点云，实现从2D图像到3D空间的语义映射
- 通过空间偏移向量和深度约束，提升3D物体定位精度
- 支持开集词汇物体定位，降低训练成本，适合交互式3D理解任务
3D高斯点云（3DGS）已成为高质量、实时渲染新视角合成的核心技术。然而现有方法主要关注几何与外观建模，缺乏深层场景理解，且训练成本高，破坏了原本简洁的可微渲染流程。为此，我们提出VoteSplat，一种融合霍夫投票与3DGS的新型3D场景理解框架。首先利用分割一切模型（SAM）进行实例分割，提取物体并生成2D投票图。随后将空间偏移向量嵌入高斯基元，通过关联2D图像投票与3D空间投票构建3D空间投票，同时利用深度畸变约束优化深度轴上的定位。针对开放词汇物体定位，VoteSplat通过投票点将2D图像语义映射至3D点云，避免高维CLIP特征的高昂训练开销，同时保持语义清晰。大量实验表明，VoteSplat在开放词汇3D实例定位、3D点云理解、点击式3D物体定位、层次化分割等方面表现优异，并通过消融实验验证其有效性。代码已公开于https://sy-ja.github.io/votesplat/
原文摘要 · Abstract (English)
3D Gaussian Splatting (3DGS) has become horsepower in high-quality, real-time rendering for novel view synthesis of 3D scenes. However, existing methods focus primarily on geometric and appearance modeling, lacking deeper scene understanding while also incurring high training costs that complicate the originally streamlined differentiable rendering pipeline. To this end, we propose VoteSplat, a novel 3D scene understanding framework that integrates Hough voting with 3DGS. Specifically, Segment Anything Model (SAM) is utilized for instance segmentation, extracting objects, and generating 2D vote maps. We then embed spatial offset vectors into Gaussian primitives. These offsets construct 3D spatial votes by associating them with 2D image votes, while depth distortion constraints refine localization along the depth axis. For open-vocabulary object localization, VoteSplat maps 2D image semantics to 3D point clouds via voting points, reducing training costs associated with high-dimensional CLIP features while preserving semantic unambiguity. Extensive experiments demonstrate effectiveness of VoteSplat in open-vocabulary 3D instance localization, 3D point cloud understanding, click-based 3D object localization, hierarchical segmentation, and ablation studies. Our code is available at https://sy-ja.github.io/votesplat/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。