用大模型生成新图像，提升弱监督分割性能
Image Augmentation Agent for Weakly Supervised Semantic Segmentation
- 用大语言模型与扩散模型自动生成训练图像
- 在PASCAL VOC和COCO数据集上超越现有方法
- 适合关注数据增强与弱监督学习的研究者
弱监督语义分割（WSSS）仅使用图像级标签已取得显著进展。然而，多数现有方法聚焦于设计新型网络结构和损失函数以生成更精确的密集标签，忽视了固定数据集带来的局限性，限制了性能提升。本文认为，更多样化的可训练图像能为WSSS提供更丰富的信息，帮助模型理解更全面的语义模式。为此，我们提出一种名为图像增强代理（IAA）的新方法，证明从数据生成角度增强WSSS是可行的。IAA设计了一个增强代理，利用大语言模型（LLMs）和扩散模型自动生成额外图像用于训练。为解决LLM提示生成不稳定的问题，我们引入提示自精炼机制，使LLM能重新评估生成提示的合理性，从而产生更连贯的提示。此外，我们在扩散生成过程中插入在线过滤器，动态保障生成图像的质量与类别平衡。实验结果表明，该方法在PASCAL VOC 2012和MS COCO 2014数据集上显著优于当前最优的WSSS方法。
原文摘要 · Abstract (English)
Weakly-supervised semantic segmentation (WSSS) has achieved remarkable progress using only image-level labels. However, most existing WSSS methods focus on designing new network structures and loss functions to generate more accurate dense labels, overlooking the limitations imposed by fixed datasets, which can constrain performance improvements. We argue that more diverse trainable images provides WSSS richer information and help model understand more comprehensive semantic pattern. Therefore in this paper, we introduce a novel approach called Image Augmentation Agent (IAA) which shows that it is possible to enhance WSSS from data generation perspective. IAA mainly design an augmentation agent that leverages large language models (LLMs) and diffusion models to automatically generate additional images for WSSS. In practice, to address the instability in prompt generation by LLMs, we develop a prompt self-refinement mechanism. It allow LLMs to re-evaluate the rationality of generated prompts to produce more coherent prompts. Additionally, we insert an online filter into diffusion generation process to dynamically ensure the quality and balance of generated images. Experimental results show that our method significantly surpasses state-of-the-art WSSS approaches on the PASCAL VOC 2012 and MS COCO 2014 datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。