用标签扩散生成分割图,让语言驱动分割更精准。
GS: Generative Segmentation via Label Diffusion
- 将分割视为生成任务,从噪声直接生成掩码。
- 在全景叙事定位任务上超越现有方法,达新最好水平。
- 适合需要精细语义与空间对齐的研究者。
语言驱动图像分割是视觉-语言理解的核心任务,要求模型根据自然语言描述分割图像中对应区域。传统方法将其视为判别问题,基于语义对齐判断像素是否为前景。近期扩散模型被引入该领域,但现有方法仍以图像为中心:或使用图像扩散模型提取视觉特征,或通过图像生成合成分割数据训练判别模型,或利用扩散反演从预训练模型中提取注意力线索——均将分割作为辅助过程。本文提出GS(Generative Segmentation),首次将分割本身建模为生成任务,通过标签扩散实现。不同于条件生成图像,GS反向过程直接从噪声生成分割掩码,同时依赖输入图像与语言描述。该范式使标签生成成为核心目标,支持端到端训练并显式控制空间与语义保真度。我们在代表性的多模态分割基准全景叙事定位(PNG)上验证了方法有效性,结果表明GS显著优于现有判别与扩散基方法,达到语言驱动分割新最佳性能。
原文摘要 · Abstract (English)
Language-driven image segmentation is a fundamental task in vision-language understanding, requiring models to segment regions of an image corresponding to natural language expressions. Traditional methods approach this as a discriminative problem, assigning each pixel to foreground or background based on semantic alignment. Recently, diffusion models have been introduced to this domain, but existing approaches remain image-centric: they either (i) use image diffusion models as visual feature extractors, (ii) synthesize segmentation data via image generation to train discriminative models, or (iii) perform diffusion inversion to extract attention cues from pre-trained image diffusion models-thereby treating segmentation as an auxiliary process. In this paper, we propose GS (Generative Segmentation), a novel framework that formulates segmentation itself as a generative task via label diffusion. Instead of generating images conditioned on label maps and text, GS reverses the generative process: it directly generates segmentation masks from noise, conditioned on both the input image and the accompanying language description. This paradigm makes label generation the primary modeling target, enabling end-to-end training with explicit control over spatial and semantic fidelity. To demonstrate the effectiveness of our approach, we evaluate GS on Panoptic Narrative Grounding (PNG), a representative and challenging benchmark for multimodal segmentation that requires panoptic-level reasoning guided by narrative captions. Experimental results show that GS significantly outperforms existing discriminative and diffusion-based methods, setting a new state-of-the-art for language-driven segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。