arXiv:2409.10389cs.CV2024-09TPAMI被引 34

让模型像人一样聚焦目标物体,提升少样本分割泛化能力

Prompt-and-Transfer: Dynamic Class-aware Enhancement for Few-shot Segmentation

论文配图:Prompt-and-Transfer: Dynamic Class-aware Enhancement for Few-shot Segmentation
图 1 · 摘自论文原文
  • 用动态提示机制引导编码器关注目标类别
  • 在11个基准上刷新少样本分割性能纪录
  • 适合跨域、弱标注及零样本分割场景

为更高效地泛化到未见领域(类别),当前多数少样本分割(FSS)方法直接使用预训练编码器并仅微调解码器。然而,这类固定特征编码器往往缺乏类别感知,会激活与目标类别无关的物体。相比之下,人类能轻松聚焦视线中的特定物体。本文模仿人类视觉感知模式,提出一种新颖强大的提示驱动方案——Prompt and Transfer(PAT),构建动态类别感知提示范式,使编码器在当前任务中聚焦于感兴趣的物体(目标类别)。三个关键设计增强提示效果:1)引入跨模态语言信息初始化每个任务的提示;2)语义提示迁移(SPT)精准传递图像内类特定语义至提示;3)部件掩码生成器(PMG)与SPT协同工作,自适应生成不同但互补的部件提示以应对个体差异。令人惊讶的是,PAT在4项不同任务上表现优异,包括标准少样本分割、跨域少样本分割(如计算机视觉、医学、遥感领域)、弱标签少样本分割和零样本分割,在11个基准上均达到新最优性能。

原文摘要 · Abstract (English)

For more efficient generalization to unseen domains (classes), most Few-shot Segmentation (FSS) would directly exploit pre-trained encoders and only fine-tune the decoder, especially in the current era of large models. However, such fixed feature encoders tend to be class-agnostic, inevitably activating objects that are irrelevant to the target class. In contrast, humans can effortlessly focus on specific objects in the line of sight. This paper mimics the visual perception pattern of human beings and proposes a novel and powerful prompt-driven scheme, called ``Prompt and Transfer" (PAT), which constructs a dynamic class-aware prompting paradigm to tune the encoder for focusing on the interested object (target class) in the current task. Three key points are elaborated to enhance the prompting: 1) Cross-modal linguistic information is introduced to initialize prompts for each task. 2) Semantic Prompt Transfer (SPT) that precisely transfers the class-specific semantics within the images to prompts. 3) Part Mask Generator (PMG) that works in conjunction with SPT to adaptively generate different but complementary part prompts for different individuals. Surprisingly, PAT achieves competitive performance on 4 different tasks including standard FSS, Cross-domain FSS (e.g., CV, medical, and remote sensing domains), Weak-label FSS, and Zero-shot Segmentation, setting new state-of-the-arts on 11 benchmarks.

少样本分割提示学习跨域泛化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。