用扩散模型实现快速精准的开放词汇分割，一次处理所有类别。
FA-Seg: A Fast and Accurate Diffusion-Based Method for Open-Vocabulary Segmentation
- 基于预训练扩散模型，仅需(1+1)步完成分割，无需训练。
- 多分辨率注意力融合提升语义精度，平均mIoU达43.8%。
- 支持零样本、全类一次性分割，适合高效部署场景。
开放词汇语义分割（OVSS）旨在对任意文本类别进行分割，而无需密集标注数据集。尽管基于对比学习的模型可实现零样本分割，但常因全局表征偏差导致像素级空间精度下降。相比之下，扩散模型通过注意力机制天然捕捉全局上下文与局部细节，具备精细空间特征表达能力。然而，其在计算成本与分割质量之间难以平衡。本文提出FA-Seg，一种基于扩散模型的训练无关框架，实现快速且准确的开放词汇分割。FA-Seg仅需从预训练扩散模型中执行(1+1)步推理，并能一次性完成所有类别的分割。为提升分割质量，引入三项关键组件：(i) 双提示机制以提取判别性、类别感知的注意力；(ii) 分层注意力精炼方法（HARD），通过多分辨率注意力融合增强语义精度；(iii) 测试时翻转（TTF）策略，改善空间一致性。大量实验表明，FA-Seg在PASCAL VOC、PASCAL Context和COCO Object基准上达到43.8%的平均mIoU，处于当前最优训练无关性能水平，同时保持优异推理效率。结果验证了其在扩展性上的潜力，弥合了分割质量与推理效率之间的差距。
原文摘要 · Abstract (English)
Open-vocabulary semantic segmentation (OVSS) aims to segment objects from arbitrary text categories without requiring densely annotated datasets. Although contrastive learning based models enable zero-shot segmentation, they often lose fine spatial precision at pixel level, due to global representation bias. In contrast, diffusion-based models naturally encode fine-grained spatial features via attention mechanisms that capture both global context and local details. However, they often face challenges in balancing the computation costs and the quality of the segmentation mask. In this work, we present FA-Seg, a Fast and Accurate training-free framework for open-vocabulary segmentation based on diffusion models. FA-Seg performs segmentation using only a (1+1)-step from a pretrained diffusion model. Moreover, instead of running multiple times for different classes, FA-Seg performs segmentation for all classes at once. To further enhance the segmentation quality, FA-Seg introduces three key components: (i) a dual-prompt mechanism for discriminative, class-aware attention extraction, (ii) a Hierarchical Attention Refinement Method (HARD) that enhances semantic precision via multi-resolution attention fusion, and (iii) a Test-Time Flipping (TTF) scheme designed to improve spatial consistency. Extensive experiments show that FA-Seg achieves state-of-the-art training-free performance, obtaining 43.8% average mIoU across PASCAL VOC, PASCAL Context, and COCO Object benchmarks while maintaining superior inference efficiency. Our results demonstrate that FA-Seg provides a strong foundation for extendability, bridging the gap between segmentation quality and inference efficiency. The source code is available at https://github.com/chequanghuy/FA-Seg.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。