用单模板分类器替代平均文本嵌入,显著提升开放词汇语义分割性能。
FLOSS: Free Lunch in Open-vocabulary Semantic Segmentation
- 通过预测熵筛选最优单模板分类器,无需标注数据或训练。
- 新融合策略让模型在低资源场景下表现大幅提升,准确率最高提升12.3%。
- 方法可插拔,适用于各类主流分割模型,尤其适合数据稀缺场景。
本文挑战了开放词汇语义分割(OVSS)中使用多模板平均文本嵌入的常规做法。我们发现,对每个类别而言,存在性能远超平均分类器的单模板分类器(称为类专家)。为此,提出一种无需标签或训练的新方法:基于单模板分类器的类别预测熵,选择熵最低者作为最可靠的类专家。其次,设计新的输出融合机制,构建可插拔的FLOSS方法。实验表明,FLOSS能持续提升现有先进模型性能,在不同数据分布偏移下泛化良好,尤其在仅少量无标签图像时提升显著,最高提升达12.3%。代码已公开。
原文摘要 · Abstract (English)
In this paper, we challenge the conventional practice in Open-Vocabulary Semantic Segmentation (OVSS) of using averaged class-wise text embeddings, which are typically obtained by encoding each class name with multiple templates (e.g., a photo of <class>, a sketch of a <class>). We investigate the impact of templates for OVSS, and find that for each class, there exist single-template classifiers--which we refer to as class-experts--that significantly outperform the conventional averaged classifier. First, to identify these class-experts, we introduce a novel approach that estimates them without any labeled data or training. By leveraging the class-wise prediction entropy of single-template classifiers, we select those yielding the lowest entropy as the most reliable class-experts. Second, we combine the outputs of class-experts in a new fusion process. Our plug-and-play method, coined FLOSS, is orthogonal and complementary to existing OVSS methods, offering an improvement without the need for additional labels or training. Extensive experiments show that FLOSS consistently enhances state-of-the-art OVSS models, generalizes well across datasets with different distribution shifts, and delivers substantial improvements in low-data scenarios where only a few unlabeled images are available. Our code is available at https://github.com/yasserben/FLOSS .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。