无需训练,用文本引导生成精准分割掩码,解决开放集语义分割难题。
Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation
- 利用冻结扩散模型的注意力图生成初始掩码提示,实现零训练分割。
- 在Pascal Context 59上达52.5 mIoU(相对提升6.8%),gRefCOCO上67.73 cIoU。
- 通过语言结构约束优化视觉提示,适合多表达式、复杂场景下的开放集分割。
开放集图像分割面临巨大挑战,因现有方法通常需大量训练或微调,且难以一致地分割多样文本描述中的统一物体。为此,我们提出Segment Anyword,一种无需训练的视觉概念提示学习方法,基于冻结扩散模型的词级别交叉注意力图生成分割代理或掩码提示,并将其优化为目标物体掩码。初始提示常因图像-文本复杂度增加而缺乏连贯性与一致性,导致分割碎片化。为此,我们引入一种语言引导的视觉提示正则化方法,依据句子依存关系与句法结构绑定并聚类视觉提示,从而提取鲁棒、抗噪的掩码提示,显著提升分割精度。该方法在多种开放集分割任务中表现优异,在Pascal Context 59上达到52.5 mIoU(相对提升6.8%),gRefCOCO上67.73 cIoU(相对提升25.73%),GranDf(最复杂开放集任务)上67.4 mIoU(优于微调方法1.1%),为当前最优结果。
原文摘要 · Abstract (English)
Open-set image segmentation poses a significant challenge because existing methods often demand extensive training or fine-tuning and generally struggle to segment unified objects consistently across diverse text reference expressions. Motivated by this, we propose Segment Anyword, a novel training-free visual concept prompt learning approach for open-set language grounded segmentation that relies on token-level cross-attention maps from a frozen diffusion model to produce segmentation surrogates or mask prompts, which are then refined into targeted object masks. Initial prompts typically lack coherence and consistency as the complexity of the image-text increases, resulting in suboptimal mask fragments. To tackle this issue, we further introduce a novel linguistic-guided visual prompt regularization that binds and clusters visual prompts based on sentence dependency and syntactic structural information, enabling the extraction of robust, noise-tolerant mask prompts, and significant improvements in segmentation accuracy. The proposed approach is effective, generalizes across different open-set segmentation tasks, and achieves state-of-the-art results of 52.5 (+6.8 relative) mIoU on Pascal Context 59, 67.73 (+25.73 relative) cIoU on gRefCOCO, and 67.4 (+1.1 relative to fine-tuned methods) mIoU on GranDf, which is the most complex open-set grounded segmentation task in the field.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。