arXiv:2506.22979cs.CV2025-06ICCV被引 4

用概率化方式优化视觉语言模型原型,提升少样本分割泛化能力

Probabilistic Prototype Calibration of Vision-Language Models for Generalized Few-shot Semantic Segmentation

  • 引入可学习的视觉校准原型,动态调整冻结文本原型
  • 通过分布正则化实现不确定性感知,减少新类过拟合
  • 在PASCAL-5i和COCO-20i上超越现有方法,适合少样本分割研究者

通用少样本语义分割(GFSS)旨在仅用少量标注样本扩展分割模型至新类别,同时保持对基础类别的性能。近期,预训练视觉语言模型(如CLIP)被用于GFSS,通过多模态原型学习提升新类泛化能力。然而,现有原型方法本质为确定性,限制了原型对多样样本的适应性,尤其在标注稀缺的新类别上。为此,我们提出FewCLIP,一种基于预训练CLIP多模态原型的概率化原型校准框架,实现更自适应的原型学习。具体而言,FewCLIP引入原型校准机制,用可学习的视觉校准原型优化冻结的文本原型,获得更具区分性的表示。此外,不同于确定性方法,FewCLIP对校准原型引入分布正则化,实现结构化且具备不确定性感知的原型学习,有效缓解对有限新类数据的过拟合,增强泛化能力。在PASCAL-5$^i$和COCO-20$^i$数据集上的大量实验表明,所提FewCLIP在GFSS及类别增量设置下均显著优于当前最优方法。代码已开源。

原文摘要 · Abstract (English)

Generalized Few-Shot Semantic Segmentation (GFSS) aims to extend a segmentation model to novel classes with only a few annotated examples while maintaining performance on base classes. Recently, pretrained vision-language models (VLMs) such as CLIP have been leveraged in GFSS to improve generalization on novel classes through multi-modal prototypes learning. However, existing prototype-based methods are inherently deterministic, limiting the adaptability of learned prototypes to diverse samples, particularly for novel classes with scarce annotations. To address this, we propose FewCLIP, a probabilistic prototype calibration framework over multi-modal prototypes from the pretrained CLIP, thus providing more adaptive prototype learning for GFSS. Specifically, FewCLIP first introduces a prototype calibration mechanism, which refines frozen textual prototypes with learnable visual calibration prototypes, leading to a more discriminative and adaptive representation. Furthermore, unlike deterministic prototype learning techniques, FewCLIP introduces distribution regularization over these calibration prototypes. This probabilistic formulation ensures structured and uncertainty-aware prototype learning, effectively mitigating overfitting to limited novel class data while enhancing generalization. Extensive experimental results on PASCAL-5$^i$ and COCO-20$^i$ datasets demonstrate that our proposed FewCLIP significantly outperforms state-of-the-art approaches across both GFSS and class-incremental setting. The code is available at https://github.com/jliu4ai/FewCLIP.

少样本分割视觉语言模型原型学习概率建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。