arXiv:2503.23806cs.CV2025-03

通过拆解物体的部件与状态,实现视觉与语言理解的精准匹配。

Bridge the Gap Between Visual and Linguistic Comprehension for Generalized Zero-shot Semantic Segmentation

  • 将物体分解为部件和状态,从视觉与语言双视角匹配
  • 在PASCAL VOC、COCO-Stuff等数据集上超越现有方法
  • 适合关注细粒度语义分割与跨模态对齐的研究者

通用零样本语义分割(GZS3)旨在通过语义表示(如词向量)的桥梁,实现对训练中未见类别的分割能力。然而,仅依赖单一语义表示进行类间关联,既不充分也不符合人类认知。受人类常利用‘部件’和‘状态’信息理解已知对象并想象未知类别的启发,本文将每类物体解耦为部件与状态的详细描述。基于此,提出解耦视觉-语言匹配(DeVLMatch)框架,包含空间部件(SPMatch)与通道状态(CSMatch)匹配模块:SPMatch从视觉与语言双视角理解物体部件并进行图匹配;CSMatch将语言中的物体状态与视觉通道信息匹配。通过在部件与状态层面解耦与匹配,可显式分析已知与未知类间的细粒度关系,促进视觉空间中的知识迁移。DeVLMatch在标准基准(PASCAL VOC、COCO-Stuff、CATARACTS)上优于先前方法,验证了其有效性。

原文摘要 · Abstract (English)

Generalized zero-shot semantic segmentation (GZS3) aims to achieve the human-level capability of segmenting not only seen classes but also novel class regions unseen in the training data through introducing the bridge of semantic representations, e.g., word vector. While effective, the way of utilizing one semantic representation to associate the corresponding class and to enable the knowledge transfer from seen to unseen classes is insufficient as well as incompatible with human cognition. Inspired by the observation that humans often use some `part' and `state' information to comprehend the seen objects and imagine unseen classes, we decouple each class into detailed descriptions, including object parts and states. Based on the decoupling formulation, we propose a Decoupled Vision-Language Matching (DeVLMatch) framework, composed of spatial-part (SPMatch) and channel-state (CSMatch) matching modules, for GZS3. In SPMatch, we comprehend objects with spatial part information from both visual and linguistic perspectives and perform graph matching to bridge the gap. In CSMatch, states of objects from the linguistic perspective are matched to compatible channel information from the visual perspective. By decoupling and matching objects across visual and linguistic comprehension, we can explicitly introspect the relationship between seen and unseen classes in fine-grained object part and state levels, thereby facilitating the knowledge transfer from seen to unseen classes in visual space. The proposed DeVLMatch framework surpasses the previous GZS3 methods on standard benchmarks, including PASCAL VOC, COCO-Stuff, and CATARACTS, demonstrating its effectiveness.

语义分割零样本学习多模态对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。