用视觉Transformer的注意力图做弱监督分割，自动对齐类别与区域。
Know Your Attention Maps: Class-specific Token Masking for Weakly Supervised Semantic Segmentation
- 每个类别设一个[CLS]令牌，通过随机掩码训练确保注意力对应正确类别。
- 在COCO、PASCAL VOC等数据集上生成伪分割图，精度优于现有方法。
- 适合缺乏精细标注数据但需高精度分割的场景，提升模型可解释性。
弱监督语义分割（WSSS）是近年来广泛研究的难题。传统方法常依赖外部模块如类别激活图来定位感兴趣区域并生成伪分割掩码。本文提出一种端到端方法，直接利用视觉变压器（ViT）学习的注意力图进行WSSS。我们训练一个稀疏的ViT，其中包含多个[CLS]令牌（每个类别一个），采用随机掩码策略促进[CLS]令牌与类别的对应关系。推理时，将预测标签对应的各[CLS]令牌的自注意力图聚合，生成伪分割掩码。该方法提升了自注意力图的可解释性，并确保准确的类别分配。在两个标准基准和三个专用数据集上的大量实验表明，本方法生成的伪掩码精度高于相关工作。这些伪掩码可用于训练分割模型，性能接近全监督模型，显著降低对细粒度标注数据的需求。
原文摘要 · Abstract (English)
Weakly Supervised Semantic Segmentation (WSSS) is a challenging problem that has been extensively studied in recent years. Traditional approaches often rely on external modules like Class Activation Maps to highlight regions of interest and generate pseudo segmentation masks. In this work, we propose an end-to-end method that directly utilizes the attention maps learned by a Vision Transformer (ViT) for WSSS. We propose training a sparse ViT with multiple [CLS] tokens (one for each class), using a random masking strategy to promote [CLS] token - class assignment. At inference time, we aggregate the different self-attention maps of each [CLS] token corresponding to the predicted labels to generate pseudo segmentation masks. Our proposed approach enhances the interpretability of self-attention maps and ensures accurate class assignments. Extensive experiments on two standard benchmarks and three specialized datasets demonstrate that our method generates accurate pseudo-masks, outperforming related works. Those pseudo-masks can be used to train a segmentation model which achieves results comparable to fully-supervised models, significantly reducing the need for fine-grained labeled data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。