arXiv:2609.06491cs.CV2026-09

用扩散模型自动生成任意物体分割掩码,无需人工标注。

Diffuse2Seg: Diffusion Models Can Segment Anything Without Supervision

论文配图:Diffuse2Seg: Diffusion Models Can Segment Anything Without Supervision
图 1 · 摘自论文原文
  • 利用扩散模型的注意力特征,通过点提示传播生成分割图。
  • 在五个数据集上比现有方法提升4.3至7.1个百分点,平均精度达90.2%
  • 适合零样本、弱监督及半监督场景下的开放世界分割任务。

开放世界实体分割旨在跨域、多粒度地预测任意物体的掩码,从部件到完整对象。当前标准SAM基于SA-1B数据集(1100万张图像,超10亿个精细标注掩码)训练,表现优异但标注成本极高。本文提出Diffuse2Seg,借助文本到图像扩散模型的中间特征,利用其在去噪过程中自然蕴含的物体结构信息,实现无监督自动掩码生成。该方法通过边缘保持方式,将网格化点提示沿扩散模型的自注意力表示传播,生成多粒度实例掩码。在五个领域上,其AR_1000指标优于先前最优标签生成器4.3–7.1个百分点。使用这些生成掩码训练实例分割模型后,检测器无关的开放世界分割性能在'things'和'stuff+things'数据集上分别提升7.4和7.7个百分点,并在'stuff+things'上超越基于检测器的UnSAM 2.1个百分点。此外,基于Diffuse2Seg生成标签训练的模型,仅需5000张标注图像即超越全监督基线,展现强大半监督初始化能力。

原文摘要 · Abstract (English)

Open-world entity segmentation aims to predict masks for arbitrary objects across domains and at multiple granularities, from parts to whole objects. In this setting, SAM sets a strong standard: trained on SA-1B, comprising 11M images and over 1B carefully annotated masks, it achieves remarkable zero-shot performance. Collecting such labels is expensive and time-consuming, however, which limits how far this recipe can scale. Text-to-image diffusion models offer a way around this. Their intermediate features transfer well across perception tasks, and since object structure emerges as the model denoises a noise sample into an image conditioned on a text prompt, that structure is already encoded in these representations. They can therefore be exploited for open-world entity segmentation without retraining or supervision. Building on this observation, we introduce Diffuse2Seg, which repurposes generative diffusion models for automatic mask generation by propagating a grid of point prompts through their self-attention representations in an edge-preserving manner. Diffuse2Seg produces multi-granular instance masks and outperforms prior state-of-the-art label generators by 4.3-7.1 p.p. in AR_1000 across five domains. Training an instance segmentation model on these generated masks advances detector-free open-world segmentation by 7.4 and 7.7 p.p. on "things" and "stuff+things" datasets and surpasses the detector-based UnSAM on "stuff+things" by 2.1 p.p. in AR_1000. Finally, we show that a model trained on Diffuse2Seg labels provides a strong initialization for semi-supervised learning, outperforming its fully supervised counterpart with already 5k labeled images.

扩散模型无监督分割开放世界

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。