从信息论出发，让SAM模型更懂专业场景的分割
InfoSAM: Fine-Tuning the Segment Anything Model from An Information-Theoretic Perspective
- 用互信息约束知识蒸馏，保留预训练模型中的通用特征
- 在多个真实任务上提升SAM在专业场景的表现
- 适合需要高效微调的视觉分割应用
Segment Anything Model（SAM）作为视觉基础模型，在通用任务中表现出色，但在特定领域表现不佳。参数高效微调（PEFT）是释放SAM潜力的有效方法，但现有方法忽视了预训练模型中蕴含的域不变关系。为此，我们提出InfoSAM，一种基于信息论的微调方法，通过提取并保留预训练模型的分割知识来增强微调效果。具体地，我们设计两个基于互信息的目标：（i）压缩从预训练SAM中提取的域不变关系，尽可能排除伪不变信息；（ii）最大化教师模型（预训练SAM）与学生模型（微调后模型）间关系知识的互信息。该方法构建了一个稳健的知识蒸馏框架，用于SAM的参数高效微调。在多个基准上的大量实验验证了InfoSAM在真实任务中提升SAM家族性能的有效性，展现出对专业场景的强适应性和优越性。
原文摘要 · Abstract (English)
The Segment Anything Model (SAM), a vision foundation model, exhibits impressive zero-shot capabilities in general tasks but struggles in specialized domains. Parameter-efficient fine-tuning (PEFT) is a promising approach to unleash the potential of SAM in novel scenarios. However, existing PEFT methods for SAM neglect the domain-invariant relations encoded in the pre-trained model. To bridge this gap, we propose InfoSAM, an information-theoretic approach that enhances SAM fine-tuning by distilling and preserving its pre-trained segmentation knowledge. Specifically, we formulate the knowledge transfer process as two novel mutual information-based objectives: (i) to compress the domain-invariant relation extracted from pre-trained SAM, excluding pseudo-invariant information as possible, and (ii) to maximize mutual information between the relational knowledge learned by the teacher (pre-trained SAM) and the student (fine-tuned model). The proposed InfoSAM establishes a robust distillation framework for PEFT of SAM. Extensive experiments across diverse benchmarks validate InfoSAM's effectiveness in improving SAM family's performance on real-world tasks, demonstrating its adaptability and superiority in handling specialized scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。