用文本嵌入增强图像语义理解,实现更精准的弱监督分割。
DIAL: Dense Image-text ALignment for Weakly Supervised Semantic Segmentation

- 通过双层对齐策略融合图文信息,提升全局与局部语义感知。
- 在PASCAL VOC和MS COCO上达到新最优,提升显著且为单阶段流程。
- 适合研究弱监督分割与多模态对齐的学者参考。
弱监督语义分割(WSSS)方法通常依赖类别激活图(CAMs)生成初始种子,但受限于图像级标签的粗粒度监督,难以捕捉全局上下文。为此,我们提出DALNet——密集对齐学习网络,利用文本嵌入增强不同粒度下对物体的全面理解和精确定位。核心思想是采用双层对齐策略:(1)全局隐式对齐(GIA),通过最大化类别标记与对应文本嵌入的相似性,同时最小化与背景嵌入的相似性,以捕获全局语义;(2)局部显式对齐(LEA),利用块标记的空间信息改进目标定位。此外,我们提出一种跨模态对比学习方法,在图像与文本模态间对齐前景特征,同时将其与背景分离,从而促进缺失区域的激活并抑制干扰。在PASCAL VOC和MS COCO数据集上的大量实验表明,DALNet显著优于现有最先进的WSSS方法。该方法尤其具备高效的端到端特性,作为单阶段方法运行。
原文摘要 · Abstract (English)
Weakly supervised semantic segmentation (WSSS) approaches typically rely on class activation maps (CAMs) for initial seed generation, which often fail to capture global context due to limited supervision from image-level labels. To address this issue, we introduce DALNet, Dense Alignment Learning Network that leverages text embeddings to enhance the comprehensive understanding and precise localization of objects across different levels of granularity. Our key insight is to employ a dual-level alignment strategy: (1) Global Implicit Alignment (GIA) to capture global semantics by maximizing the similarity between the class token and the corresponding text embeddings while minimizing the similarity with background embeddings, and (2) Local Explicit Alignment (LEA) to improve object localization by utilizing spatial information from patch tokens. Moreover, we propose a cross-contrastive learning approach that aligns foreground features between image and text modalities while separating them from the background, encouraging activation in missing regions and suppressing distractions. Through extensive experiments on the PASCAL VOC and MS COCO datasets, we demonstrate that DALNet significantly outperforms state-of-the-art WSSS methods. Our approach, in particular, allows for more efficient end-to-end process as a single-stage method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。