用语言引导实现跨类别3D形状的精准匹配。
Universal 3D Shape Matching via Coarse-to-Fine Language Guidance
- 先语义分割再用大模型命名,分步建立粗粒度对应关系。
- 在非等距形状上实现高精度稠密对应,优于现有方法。
- 无需预设部件,适合任意类别物体,通用性强。
建立形状间的稠密对应是计算机视觉与图形学中的关键任务,但以往方法依赖近等距假设且仅适用于同类型物体(如仅限人体)。对于跨类别物体的语义对应仍具挑战,研究较少。为此,我们提出UniMatch,一种语义感知的、从粗到细的框架,可在不限制物体类别的前提下,为强非等距形状构建稠密语义对应。核心思想是将粗粒度语义提示提升为细粒度对应:第一阶段进行无类别依赖的3D分割,获得不重叠的语义部分,并通过多模态大语言模型(MLLM)识别部件名称;随后利用预训练视觉语言模型(VLM)提取文本嵌入,构建匹配的语义部分。第二阶段则基于这些粗对应,通过专用的基于排名的对比学习机制引导稠密对应的学习。得益于无类别分割、语言引导和基于排名的对比学习,该方法具备跨类别通用性,无需预定义部件提议,可实现跨类、非等距形状的通用匹配。大量实验表明,UniMatch在多种挑战性场景中持续优于现有方法。
原文摘要 · Abstract (English)
Establishing dense correspondences between shapes is a crucial task in computer vision and graphics, while prior approaches depend on near-isometric assumptions and homogeneous subject types (i.e., only operate for human shapes). However, building semantic correspondences for cross-category objects remains challenging and has received relatively little attention. To achieve this, we propose UniMatch, a semantic-aware, coarse-to-fine framework for constructing dense semantic correspondences between strongly non-isometric shapes without restricting object categories. The key insight is to lift "coarse" semantic cues into "fine" correspondence, which is achieved through two stages. In the "coarse" stage, we perform class-agnostic 3D segmentation to obtain non-overlapping semantic parts and prompt multimodal large language models (MLLMs) to identify part names. Then, we employ pretrained vision language models (VLMs) to extract text embeddings, enabling the construction of matched semantic parts. In the "fine" stage, we leverage these coarse correspondences to guide the learning of dense correspondences through a dedicated rank-based contrastive scheme. Thanks to class-agnostic segmentation, language guiding, and rank-based contrastive learning, our method is versatile for universal object categories and requires no predefined part proposals, enabling universal matching for inter-class and non-isometric shapes. Extensive experiments demonstrate UniMatch consistently outperforms competing methods in various challenging scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。