提升图文在像素级的对齐精度,实现更精准的开放词汇语义分割。
FGAseg: Fine-Grained Pixel-Text Alignment for Open-Vocabulary Semantic Segmentation
- 通过交叉注意力与文本-像素对齐损失,细化CLIP的粗粒度对齐
- 引入可学习伪掩码,补充类别边界信息,提升分割精度
- 适合需要细粒度图像理解与开放词汇识别的研究与应用
开放词汇语义分割旨在根据文本描述识别并分割特定区域与物体。现有方法常依赖于如CLIP等视觉语言模型(VLM)来连接视觉与文本信息,但这些模型通常在图像级对齐上预训练,关注全局语义特征,难以满足分割任务所需的像素级精细对齐与类别边界细节。直接从VLM提取的信息无法满足分割需求。为此,本文提出FGAseg,一种专注于像素级图文对齐与类别边界补全的模型。其核心是像素级对齐模块,采用跨模态注意力机制与文本-像素对齐损失,优化来自CLIP的粗粒度对齐,实现更精细的像素-文本语义对齐。此外,为增强类别边界信息,引入可学习的对齐矩阵作为前向传播中的伪掩码,并设计类别信息补全模块。这些伪掩码基于余弦相似度与卷积相似性,提供不同类别间的全局与局部边界信息。通过结合两种策略,FGAseg有效提升了像素级对齐精度与类别边界感知能力。大量实验表明,该方法在开放词汇语义分割基准上优于现有方法。
原文摘要 · Abstract (English)
Open-vocabulary segmentation aims to identify and segment specific regions and objects based on text-based descriptions. A common solution is to leverage powerful vision-language models (VLMs), such as CLIP, to bridge the gap between vision and text information. However, VLMs are typically pretrained for image-level vision-text alignment, focusing on global semantic features. In contrast, segmentation tasks require fine-grained pixel-level alignment and detailed category boundary information, which VLMs alone cannot provide. As a result, information extracted directly from VLMs can't meet the requirements of segmentation tasks. To address this limitation, we propose FGAseg, a model designed for fine-grained pixel-text alignment and category boundary supplementation. The core of FGAseg is a Pixel-Level Alignment module that employs a cross-modal attention mechanism and a text-pixel alignment loss to refine the coarse-grained alignment from CLIP, achieving finer-grained pixel-text semantic alignment. Additionally, to enrich category boundary information, we introduce the alignment matrices as optimizable pseudo-masks during forward propagation and propose Category Information Supplementation module. These pseudo-masks, derived from cosine and convolutional similarity, provide essential global and local boundary information between different categories. By combining these two strategies, FGAseg effectively enhances pixel-level alignment and category boundary information, addressing key challenges in open-vocabulary segmentation. Extensive experiments demonstrate that FGAseg outperforms existing methods on open-vocabulary semantic segmentation benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。