arXiv:2412.04247cs.CV2024-12被引 15

融合2D视觉与3D几何结构,实现无需标注的通用3D部件分割

3D Part Segmentation via Geometric Aggregation of 2D Visual Features

  • 多视角渲染提取2D特征,通过几何感知聚合映射回3D空间
  • 在5个数据集上零样本性能达顶尖水平,覆盖真实与合成场景
  • 适合需要跨物体、无标注训练的3D理解任务开发者

监督式3D部件分割模型通常针对固定物体和部件设计,难以迁移至开放场景。近期研究尝试使用视觉语言模型(VLM)结合多视角渲染与文本提示来识别部件,但存在提示工程繁琐、未利用3D几何结构等问题。为此,我们提出COPS——一种综合视觉语义与3D几何信息的部件分割模型。COPS从多个视角渲染点云,提取2D特征并投影回3D空间,采用新颖的几何感知特征聚合机制,确保空间与语义一致性。最终通过聚类将点分配至部件并打标。实验表明,COPS高效可扩展,在五个数据集上实现零样本领先性能,涵盖合成与真实数据、无纹理与彩色物体、刚性与非刚性形状。代码已公开于https://3d-cops.github.io。

原文摘要 · Abstract (English)

Supervised 3D part segmentation models are tailored for a fixed set of objects and parts, limiting their transferability to open-set, real-world scenarios. Recent works have explored vision-language models (VLMs) as a promising alternative, using multi-view rendering and textual prompting to identify object parts. However, naively applying VLMs in this context introduces several drawbacks, such as the need for meticulous prompt engineering, and fails to leverage the 3D geometric structure of objects. To address these limitations, we propose COPS, a COmprehensive model for Parts Segmentation that blends the semantics extracted from visual concepts and 3D geometry to effectively identify object parts. COPS renders a point cloud from multiple viewpoints, extracts 2D features, projects them back to 3D, and uses a novel geometric-aware feature aggregation procedure to ensure spatial and semantic consistency. Finally, it clusters points into parts and labels them. We demonstrate that COPS is efficient, scalable, and achieves zero-shot state-of-the-art performance across five datasets, covering synthetic and real-world data, texture-less and coloured objects, as well as rigid and non-rigid shapes. The code is available at https://3d-cops.github.io.

3D分割视觉语言模型几何感知零样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。