arXiv:2412.06418cs.CV2024-12被引 6

提出新方法让SAM模型持续学习,适应动态数据流

Continual Learning for Segment Anything Model Adaptation

  • 用领域适配器混合机制分离不同任务特征
  • 在8个任务域上持续学习后仍接近原始SAM零样本表现
  • 可无缝集成到现有SAM适配方法中提升性能

尽管当前基于提示或适配器的SAM适配方法在各类下游任务中表现良好,但大多属于单步适配范式。现实中数据常以流式方式动态出现,为此本文首次构建了包含8个不同任务域的持续SAM适配(CoSAM)基准,分析了现有单步适配方法在持续分割场景下的局限性。提出简单有效的领域适配器混合(MoDA)算法,利用全局特征令牌(GFT)和全局辅助令牌(GAT)模块,帮助SAM编码器提取区分度高的任务特征,并提供精准的任务特定信息以支持持续学习。大量实验表明,所提MoDA显著优于现有经典持续学习方法,以及基于提示和适配器的方法。在跨多样化数据分布的序列学习后,其在自然图像域仍保持高竞争力,接近原始SAM的零样本性能,展现出卓越的知识保留能力。值得注意的是,MoDA可无缝集成至多种单步适配方法,持续带来显著性能提升。代码已公开于https://github.com/yangjl1215/CoSAM。

原文摘要 · Abstract (English)

Although the current different types of SAM adaptation methods have achieved promising performance for various downstream tasks, such as prompt-based ones and adapter-based ones, most of them belong to the one-step adaptation paradigm. In real-world scenarios, we are generally confronted with the dynamic scenario where the data comes in a streaming manner. Driven by the practical need, in this paper, we first propose a novel Continual SAM adaptation (CoSAM) benchmark with 8 different task domains and carefully analyze the limitations of the existing SAM one-step adaptation methods in the continual segmentation scenario. Then we propose a novel simple-yet-effective Mixture of Domain Adapters (MoDA) algorithm which utilizes the Global Feature Tokens (GFT) and Global Assistant Tokens (GAT) modules to help the SAM encoder extract well-separated features for different task domains, and then provide the accurate task-specific information for continual learning. Extensive experiments demonstrate that our proposed MoDA obviously surpasses the existing classic continual learning methods, as well as prompt-based and adapter-based approaches for continual segmentation. Moreover, after sequential learning on the CoSAM benchmark with diverse data distributions, our MoDA maintains highly competitive results in the natural image domain, approaching the zero-shot performance of the original SAM, demonstrating its superior capability in knowledge preservation. Notably, the proposed MoDA can be seamlessly integrated into various one-step adaptation methods of SAM, which can consistently bring obvious performance gains. Code is available at \url{https://github.com/yangjl1215/CoSAM}

持续学习SAM图像分割适配器

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。