用2D掩码引导梯度投票，实现3D高斯点云精准分割与功能迁移
Gradient-Driven 3D Segmentation and Affordance Transfer in Gaussian Splatting Using 2D Masks
- 通过2D掩码过滤梯度，以梯度作为投票实现3D分割
- 推理时梯度可压缩21%的高斯点，提升效率
- 支持少量样本下从2D图像迁移物体功能到3D场景
3D高斯点云已成为高效捕捉精细结构的3D场景表示方法。本文提出一种新颖的基于投票的方法，将2D分割模型扩展至3D高斯点云。该方法利用被2D掩码过滤的梯度，以梯度作为投票实现精确分割。作为副产品，我们发现推理阶段的梯度可用于剪枝高斯点，实现最高达21%的压缩。此外，我们探索了少样本下的功能迁移，使2D图像中的标注可有效传递至3D高斯点云。该方法基于稳健且简洁的数学框架，适用于增强现实（AR）、物体编辑和机器人等下游应用。项目代码及资源见 https://jojijoseph.github.io/3dgs-segmentation。
原文摘要 · Abstract (English)
3D Gaussian Splatting has emerged as a powerful 3D scene representation technique, capturing fine details with high efficiency. In this paper, we introduce a novel voting-based method that extends 2D segmentation models to 3D Gaussian splats. Our approach leverages masked gradients, where gradients are filtered by input 2D masks, and these gradients are used as votes to achieve accurate segmentation. As a byproduct, we discovered that inference-time gradients can also be used to prune Gaussians, resulting in up to 21% compression. Additionally, we explore few-shot affordance transfer, allowing annotations from 2D images to be effectively transferred onto 3D Gaussian splats. The robust yet straightforward mathematical formulation underlying this approach makes it a highly effective tool for numerous downstream applications, such as augmented reality (AR), object editing, and robotics. The project code and additional resources are available at https://jojijoseph.github.io/3dgs-segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。