用基础模型隐含知识提升少样本分割性能，效果显著超越现有方法。
High-Performance Few-Shot Segmentation with Foundation Models: An Empirical Study
- 从基础模型提取隐含知识构建粗对应关系，再用轻量解码器细化分割。
- 在COCO-20i上相比之前最优方法提升17.5%。
- 适合关注少样本分割与视觉基础模型应用的研究者。
现有少样本分割（FSS）方法多聚焦于设计新颖的支持-查询匹配与自匹配机制，以利用预训练主干网络中的隐含知识。然而，这些方法的性能常受限于仅在分类任务上预训练的模型。对于哪些类型的预训练模型能为FSS提供更优隐含知识的探索仍有限。本文受视觉基础模型表示一致性启发，提出基于基础模型的FSS框架。具体而言，我们提出一种简单方法，从基础模型中提取隐含知识以构建粗略对应关系，并引入轻量解码器对粗对应进行精细化处理，实现细粒度分割。我们在两个常用数据集上系统评估了多种基础模型在FSS上的表现，发现部分基础模型中的隐含知识比传统分类预训练模型更具优势。大量实验证明，该方法有效利用了基础模型的隐含知识。特别地，DINOv2与DFN组合在COCO-20i上超越先前最优方法达17.5%。代码已开源。
原文摘要 · Abstract (English)
Existing few-shot segmentation (FSS) methods mainly focus on designing novel support-query matching and self-matching mechanisms to exploit implicit knowledge in pre-trained backbones. However, the performance of these methods is often constrained by models pre-trained on classification tasks. The exploration of what types of pre-trained models can provide more beneficial implicit knowledge for FSS remains limited. In this paper, inspired by the representation consistency of foundational computer vision models, we develop a FSS framework based on foundation models. To be specific, we propose a simple approach to extract implicit knowledge from foundation models to construct coarse correspondence and introduce a lightweight decoder to refine coarse correspondence for fine-grained segmentation. We systematically summarize the performance of various foundation models on FSS and discover that the implicit knowledge within some of these models is more beneficial for FSS than models pre-trained on classification tasks. Extensive experiments on two widely used datasets demonstrate the effectiveness of our approach in leveraging the implicit knowledge of foundation models. Notably, the combination of DINOv2 and DFN exceeds previous state-of-the-art methods by 17.5% on COCO-20i. Code is available at https://github.com/DUT-CSJ/FoundationFSS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。