用语言空间统一建模物体与部件层级，实现开放词汇的实例分割。
LangHOPS: Language Grounded Hierarchical Open-Vocabulary Part Segmentation
- 基于多模态大模型，将物体-部件层次结构对齐到语言空间中。
- 在PartImageNet上比前人提升5.5% AP（同域）和4.8% AP（跨数据集）。
- 适用于零样本场景，适合需要细粒度语义理解的研究者。
我们提出LangHOPS，首个基于多模态大模型（MLLM）的开放词汇物体-部件实例分割框架。给定一张图像，LangHOPS可联合检测并分割来自开放词汇候选类别中的物体与部件实例。不同于依赖启发式或可学习视觉分组的先前方法，本方法将物体-部件层次结构在语言空间中进行定位。通过将MLLM整合进物体-部件解析流程，利用其丰富的知识与推理能力，建立层次内多粒度概念的关联。我们在多个挑战性场景下评估了LangHOPS，包括同域与跨数据集的物体-部件实例分割及零样本语义分割。在PartImageNet上，其表现超越以往方法5.5%（同域）与4.8%（跨数据集）的平均精度（AP），在ADE20K未见物体部件上的零样本任务中，也实现了2.5%的mIOU提升。消融实验进一步验证了语言对齐层次结构与MLLM驱动的部件查询优化策略的有效性。代码将公开发布。
原文摘要 · Abstract (English)
We propose LangHOPS, the first Multimodal Large Language Model (MLLM) based framework for open-vocabulary object-part instance segmentation. Given an image, LangHOPS can jointly detect and segment hierarchical object and part instances from open-vocabulary candidate categories. Unlike prior approaches that rely on heuristic or learnable visual grouping, our approach grounds object-part hierarchies in language space. It integrates the MLLM into the object-part parsing pipeline to leverage its rich knowledge and reasoning capabilities, and link multi-granularity concepts within the hierarchies. We evaluate LangHOPS across multiple challenging scenarios, including in-domain and cross-dataset object-part instance segmentation, and zero-shot semantic segmentation. LangHOPS achieves state-of-the-art results, surpassing previous methods by 5.5% Average Precision (AP) (in-domain) and 4.8% (cross-dataset) on the PartImageNet dataset and by 2.5% mIOU on unseen object parts in ADE20K (zero-shot). Ablation studies further validate the effectiveness of the language-grounded hierarchy and MLLM driven part query refinement strategy. The code will be released here.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。