让零训练扩散模型分割器更好利用生成能力,提升分割精度。
Making Training-Free Diffusion Segmentors Scale with the Generative Power
- 通过自动聚合与像素级重缩放,优化注意力图融合方式。
- 在Cityscapes等数据集上显著提升分割性能,最高提升4.2% mIoU。
- 适合追求零训练、高泛化分割方案的研究者使用。
作为强大的生成模型,文本到图像扩散模型近年来被用于判别任务。一系列研究尝试在无需额外训练的情况下,将预训练扩散模型适配为语义分割器,形成训练免费的扩散分割器。这些方法通常依赖模型注意力层中的交叉注意力图,假设其能捕捉图像像素与文本标记间的语义关系。理想情况下,更强的生成能力应带来更好的分割效果,但现有方法往往无法随之提升。我们发现两个关键问题:(i) 多头多层交叉注意力图存在个体差异,缺乏统一全局表示;(ii) 即使有全局图,不同文本标记的得分不平衡也导致语义关联不准确。为此,我们提出自动聚合与像素级重缩放两项技术,有效弥合上述差距。我们在标准语义分割基准上评估方法,并进一步集成至生成式框架中,验证了性能提升与广泛应用潜力。代码已公开于 https://github.com/Darkbblue/goca。
原文摘要 · Abstract (English)
As powerful generative models, text-to-image diffusion models have recently been explored for discriminative tasks. A line of research focuses on adapting a pre-trained diffusion model to semantic segmentation without any further training, leading to training-free diffusion segmentors. These methods typically rely on cross-attention maps from the model's attention layers, which are assumed to capture semantic relationships between image pixels and text tokens. Ideally, such approaches should benefit from more powerful diffusion models, i.e., stronger generative capability should lead to better segmentation. However, we observe that existing methods often fail to scale accordingly. To understand this issue, we identify two underlying gaps: (i) cross-attention is computed across multiple heads and layers, but there exists a discrepancy between these individual attention maps and a unified global representation. (ii) Even when a global map is available, it does not directly translate to accurate semantic correlation for segmentation, due to score imbalances among different text tokens. To bridge these gaps, we propose two techniques: auto aggregation and per-pixel rescaling, which together enable training-free segmentation to better leverage generative capability. We evaluate our approach on standard semantic segmentation benchmarks and further integrate it into a generative technique, demonstrating both improved performance broad applicability. Codes are at https://github.com/Darkbblue/goca.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。