arXiv:2502.19782cs.CV2025-02被引 10

首个可对3D人体进行开放词汇语义分割的方法,支持文本提示精细分段。

Open-Vocabulary Semantic Part Segmentation of 3D Human

  • 用SAM生成2D多视角候选框,结合HumanCLIP统一视觉与文本嵌入。
  • 在多个3D人体数据集上显著超越现有最先进方法,提升明显。
  • 适配网格、点云、3D高斯泼溅等多种表示,推理高效实用。

3D人体部件分割在3D视觉与AR/VR领域仍是开放性问题。由于3D标注数据有限,传统监督分割方法难以泛化到未见形状与类别。近年来,视觉语言模型的零样本能力推动了开放世界3D分割的发展,但现有方法在3D人体上表现不佳。本文提出首个可处理3D人体的开放词汇分割方法,能基于文本提示将人体细分为指定语义部件。框架采用简单分割流程:利用SAM生成2D多视角提案,设计新型HumanCLIP模型构建视觉与文本输入的统一嵌入,相比预训练CLIP模型在人体内容上获得更准确表示。此外,提出简单有效的MaskFusion模块,无需复杂投票与分组机制,直接融合多视图特征生成3D语义掩码。解耦掩码提案与文本输入的设计显著提升单提示推理效率。实验表明,该方法在多个3D人体数据集上大幅领先当前最先进开放词汇3D分割方法。同时,方法可直接应用于网格、点云和3D高斯泼溅等多种3D表示。

原文摘要 · Abstract (English)

3D part segmentation is still an open problem in the field of 3D vision and AR/VR. Due to limited 3D labeled data, traditional supervised segmentation methods fall short in generalizing to unseen shapes and categories. Recently, the advancement in vision-language models' zero-shot abilities has brought a surge in open-world 3D segmentation methods. While these methods show promising results for 3D scenes or objects, they do not generalize well to 3D humans. In this paper, we present the first open-vocabulary segmentation method capable of handling 3D human. Our framework can segment the human category into desired fine-grained parts based on the textual prompt. We design a simple segmentation pipeline, leveraging SAM to generate multi-view proposals in 2D and proposing a novel HumanCLIP model to create unified embeddings for visual and textual inputs. Compared with existing pre-trained CLIP models, the HumanCLIP model yields more accurate embeddings for human-centric contents. We also design a simple-yet-effective MaskFusion module, which classifies and fuses multi-view features into 3D semantic masks without complex voting and grouping mechanisms. The design of decoupling mask proposals and text input also significantly boosts the efficiency of per-prompt inference. Experimental results on various 3D human datasets show that our method outperforms current state-of-the-art open-vocabulary 3D segmentation methods by a large margin. In addition, we show that our method can be directly applied to various 3D representations including meshes, point clouds, and 3D Gaussian Splatting.

3D分割开放词汇人体建模视觉语言

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。