用伪提示与迭代优化，让SAM2更好做少样本分割
Unlocking the Power of SAM 2 for Few-Shot Segmentation
- 设计伪提示生成器，兼容视频与分割任务的匹配需求
- 1次提示下在PASCAL-5i上达到4.2%的mIoU提升
- 适合需要少样本分割且想用SAM2的开发者
少样本分割（FSS）旨在仅用少量类别学习通用分割能力，但易过拟合。现有方法借助基础模型（如SAM）的知识简化训练过程。近期的SAM 2支持视频分割，其类无关匹配能力对FSS有益。然而，由于视频中前景对象身份一致，而FSS中为不同身份，直接匹配不兼容。为此，本文提出伪提示生成器，编码伪查询记忆以实现兼容匹配。但伪记忆可能不完整或混入背景特征，导致误分割。因此进一步设计迭代记忆精炼模块，融合更多前景特征，并引入支持校准的记忆注意力机制抑制背景干扰。在PASCAL-5$^i$和COCO-20$^i$上的大量实验验证了方法有效性，1次提示下相比最优基线，mIoU提升达4.2%。
原文摘要 · Abstract (English)
Few-Shot Segmentation (FSS) aims to learn class-agnostic segmentation on few classes to segment arbitrary classes, but at the risk of overfitting. To address this, some methods use the well-learned knowledge of foundation models (e.g., SAM) to simplify the learning process. Recently, SAM 2 has extended SAM by supporting video segmentation, whose class-agnostic matching ability is useful to FSS. A simple idea is to encode support foreground (FG) features as memory, with which query FG features are matched and fused. Unfortunately, the FG objects in different frames of SAM 2's video data are always the same identity, while those in FSS are different identities, i.e., the matching step is incompatible. Therefore, we design Pseudo Prompt Generator to encode pseudo query memory, matching with query features in a compatible way. However, the memories can never be as accurate as the real ones, i.e., they are likely to contain incomplete query FG, and some unexpected query background (BG) features, leading to wrong segmentation. Hence, we further design Iterative Memory Refinement to fuse more query FG features into the memory, and devise a Support-Calibrated Memory Attention to suppress the unexpected query BG features in memory. Extensive experiments have been conducted on PASCAL-5$^i$ and COCO-20$^i$ to validate the effectiveness of our design, e.g., the 1-shot mIoU can be 4.2% better than the best baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。