arXiv:2507.19808cs.CV2025-07AAAI被引 6

用扩散模型注意力机制自动生成高质量像素级标注掩码,无需训练或调参。

SeeDiff: Off-the-Shelf Seeded Mask Generation from Diffusion Models

论文配图:SeeDiff: Off-the-Shelf Seeded Mask Generation from Diffusion Models
图 1 · 摘自论文原文
  • 利用扩散模型的交叉注意力生成粗略种子区域。
  • 通过多尺度自注意力迭代扩展,实现类级别完整分割。
  • 无需训练、调优或预训练网络,开箱即用生成高精度掩码。

为实现像素级物体分类,语义分割网络需耗费大量人力进行像素级标注。现有方法借助文生图模型(如 Stable Diffusion)中的图文关系生成图像与标注掩码对,但通常仍需预训练分割网络、精细提示词调优或额外训练。本文深入分析 Stable Diffusion 的注意力机制,发现交叉注意力可提供粗略对象定位,作为初始种子;随后利用自注意力的语义对应能力,通过多尺度自注意力图从种子区域迭代扩展至整类对象。同时观察到,简单文本引导生成的图像背景均匀,更易建立对应关系,因此进一步用更精确的背景掩码优化最终结果。所提方法 SeeDiff 可直接从 Stable Diffusion 生成高质量掩码,无需额外训练、提示词调优或预训练分割网络。

原文摘要 · Abstract (English)

Entrusted with the goal of pixel-level object classification, the semantic segmentation networks entail the laborious preparation of pixel-level annotation masks. To obtain pixel-level annotation masks for a given class without human efforts, recent few works have proposed to generate pairs of images and annotation masks by employing image and text relationships modeled by text-to-image generative models, especially Stable Diffusion. However, these works do not fully exploit the capability of text-guided Diffusion models and thus require a pre-trained segmentation network, careful text prompt tuning, or the training of a segmentation network to generate final annotation masks. In this work, we take a closer look at attention mechanisms of Stable Diffusion, from which we draw connections with classical seeded segmentation approaches. In particular, we show that cross-attention alone provides very coarse object localization, which however can provide initial seeds. Then, akin to region expansion in seeded segmentation, we utilize the semantic-correspondence-modeling capability of self-attention to iteratively spread the attention to the whole class from the seeds using multi-scale self-attention maps. We also observe that a simple-text-guided synthetic image often has a uniform background, which is easier to find correspondences, compared to complex-structured objects. Thus, we further refine a mask using a more accurate background mask. Our proposed method, dubbed SeeDiff, generates high-quality masks off-the-shelf from Stable Diffusion, without additional training procedure, prompt tuning, or a pre-trained segmentation network.

扩散模型语义分割自动标注注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。