让SAM2隐含语义显式化,实现少样本分割新突破
SANSA: Unleashing the Hidden Semantics in SAM2 for Few-Shot Segmentation
- 通过轻量级改造显式挖掘SAM2的语义结构
- 在通用性测试中超越现有主流方法
- 支持点/框/草图多种交互方式,速度快体积小
少样本分割旨在仅凭少量标注样例即可分割未见类别。这需要同时具备跨图像识别语义相关对象的能力与生成精确分割掩码的能力。我们发现,具有提示-传播机制的Segment Anything 2(SAM2)虽具强大分割能力与内置特征匹配过程,但其表征受任务特定线索(如目标跟踪)干扰,影响高层语义理解。关键洞察是:尽管预训练为无类别,SAM2的特征中已蕴含丰富语义结构。我们提出SANSA(Semantically Aligned Segment Anything 2),通过最小化任务特异性修改使该隐含结构显式化,成功将SAM2重构用于少样本分割。SANSA在专为评估泛化能力设计的基准上达到当前最优性能,在流行的上下文学习设置中优于通用方法,支持点、框或草图等多种灵活提示交互,且显著更快更紧凑。代码已公开于https://github.com/ClaudiaCuttano/SANSA。
原文摘要 · Abstract (English)
Few-shot segmentation aims to segment unseen object categories from just a handful of annotated examples. This requires mechanisms that can both identify semantically related objects across images and accurately produce segmentation masks. We note that Segment Anything 2 (SAM2), with its prompt-and-propagate mechanism, offers both strong segmentation capabilities and a built-in feature matching process. However, we show that its representations are entangled with task-specific cues optimized for object tracking, which impairs its use for tasks requiring higher level semantic understanding. Our key insight is that, despite its class-agnostic pretraining, SAM2 already encodes rich semantic structure in its features. We propose SANSA (Semantically AligNed Segment Anything 2), a framework that makes this latent structure explicit, and repurposes SAM2 for few-shot segmentation through minimal task-specific modifications. SANSA achieves state-of-the-art performance on few-shot segmentation benchmarks specifically designed to assess generalization, outperforms generalist methods in the popular in-context setting, supports various prompts flexible interaction via points, boxes, or scribbles, and remains significantly faster and more compact than prior approaches. Code is available at https://github.com/ClaudiaCuttano/SANSA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。