arXiv:2511.15118cs.CV2025-11被引 1

用语义引导解码,让大模型少偏见地完成少样本分割。

Unbiased Semantic Decoding with Vision Foundation Models for Few-shot Segmentation

  • 双端提取支持集与查询集语义,通过CLIP对齐增强特征
  • 设计全局补全与局部引导策略,提升类别泛化能力
  • 无需重训练模型,适合快速适配新类别场景

少样本分割受到广泛关注。许多近期方法尝试引入图像分割通用模型(SAM)来处理该任务。由于SAM具备强大的泛化能力和丰富的对象级特征提取能力,这一方案在少样本分割中展现出巨大潜力。然而,SAM的解码过程高度依赖准确且明确的提示,以往方法主要关注从支持集中提取提示,难以充分激活SAM的泛化能力,且在适应未知类别时易导致解码偏差。为此,本文提出一种与SAM集成的无偏语义解码(USD)策略,同时从支持集和查询集中提取目标信息,并在对比语言-图像预训练(CLIP)模型的语义引导下实现一致预测。具体地,为增强SAM的无偏语义判别能力,设计了两种特征增强策略:在图像层面进行全局补充以提供通用类别指示,在像素层面进行局部引导以提供目标位置信息。此外,提出可学习的视觉-文本目标提示生成器,通过交互目标文本嵌入与CLIP视觉特征生成聚焦目标的提示嵌入。无需对视觉基础模型进行再训练,具有语义判别力的特征通过富含目标信息的提示引导,聚焦于目标区域。

原文摘要 · Abstract (English)

Few-shot segmentation has garnered significant attention. Many recent approaches attempt to introduce the Segment Anything Model (SAM) to handle this task. With the strong generalization ability and rich object-specific extraction ability of the SAM model, such a solution shows great potential in few-shot segmentation. However, the decoding process of SAM highly relies on accurate and explicit prompts, making previous approaches mainly focus on extracting prompts from the support set, which is insufficient to activate the generalization ability of SAM, and this design is easy to result in a biased decoding process when adapting to the unknown classes. In this work, we propose an Unbiased Semantic Decoding (USD) strategy integrated with SAM, which extracts target information from both the support and query set simultaneously to perform consistent predictions guided by the semantics of the Contrastive Language-Image Pre-training (CLIP) model. Specifically, to enhance the unbiased semantic discrimination of SAM, we design two feature enhancement strategies that leverage the semantic alignment capability of CLIP to enrich the original SAM features, mainly including a global supplement at the image level to provide a generalize category indicate with support image and a local guidance at the pixel level to provide a useful target location with query image. Besides, to generate target-focused prompt embeddings, a learnable visual-text target prompt generator is proposed by interacting target text embeddings and clip visual features. Without requiring re-training of the vision foundation models, the features with semantic discrimination draw attention to the target region through the guidance of prompt with rich target information.

少样本分割CLIPSAM无偏解码

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。