arXiv:2505.19569cs.CV2025-05被引 2

模仿人类认知过程,让模型先理解概念再定位图像区域。

What You Perceive Is What You Conceive: A Cognition-Inspired Framework for Open Vocabulary Image Segmentation

  • 先生成语义概念再进行分割,模拟人脑认知顺序。
  • 在A-150上达27.2 PQ、35.3 mIoU,多项指标领先。
  • 支持无词汇分割,可灵活识别未见类别,适合开放场景。

开放词汇图像分割需在推理时动态识别预定义的新类别,依赖视觉-语言对齐。现有方法通常先进行类无关区域分割,再匹配类别,偏离了人类基于语义概念识别物体的机制,导致分割区域与目标概念对齐不佳。为此,我们提出一种认知启发式框架,模拟人类视觉识别过程:先形成物体概念理解,再感知其空间范围。框架包含三个核心组件:(1) 生成式视觉-语言模型(G-VLM),通过生成对象概念为分割提供语义引导;(2) 概念感知视觉增强模块,融合文本概念特征与全局视觉表征,实现基于目标概念的自适应视觉感知;(3) 认知启发解码器,整合局部实例特征与G-VLM提供的语义线索,实现对相关类别子集的选择性分类。大量实验表明,该框架在A-150上达到27.2 PQ、17.0 mAP、35.3 mIoU;在Cityscapes、Mapillary Vistas、A-847、PC-59、PC-459和PAS-20上分别取得56.2、28.2、15.4、59.2、18.7、95.8 mIoU。此外,框架支持无词汇分割,显著提升对未知类别的识别灵活性。代码将公开。

原文摘要 · Abstract (English)

Open vocabulary image segmentation tackles the challenge of recognizing dynamically adjustable, predefined novel categories at inference time by leveraging vision-language alignment. However, existing paradigms typically perform class-agnostic region segmentation followed by category matching, which deviates from the human visual system's process of recognizing objects based on semantic concepts, leading to poor alignment between region segmentation and target concepts. To bridge this gap, we propose a novel Cognition-Inspired Framework for open vocabulary image segmentation that emulates the human visual recognition process: first forming a conceptual understanding of an object, then perceiving its spatial extent. The framework consists of three core components: (1) A Generative Vision-Language Model (G-VLM) that mimics human cognition by generating object concepts to provide semantic guidance for region segmentation. (2) A Concept-Aware Visual Enhancer Module that fuses textual concept features with global visual representations, enabling adaptive visual perception based on target concepts. (3) A Cognition-Inspired Decoder that integrates local instance features with G-VLM-provided semantic cues, allowing selective classification over a subset of relevant categories. Extensive experiments demonstrate that our framework achieves significant improvements, reaching $27.2$ PQ, $17.0$ mAP, and $35.3$ mIoU on A-150. It further attains $56.2$, $28.2$, $15.4$, $59.2$, $18.7$, and $95.8$ mIoU on Cityscapes, Mapillary Vistas, A-847, PC-59, PC-459, and PAS-20, respectively. In addition, our framework supports vocabulary-free segmentation, offering enhanced flexibility in recognizing unseen categories. Code will be public.

图像分割开放词汇认知启发多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。