用文本提示提升SAM分割性能，仅改少量参数。
SAM-PTx: Text-Guided Fine-Tuning of SAM with Parameter-Efficient, Parallel-Text Adapters
- 设计轻量级并行文本适配器，注入预训练文本嵌入
- 在COD10K等数据集上显著优于纯空间提示基线
- 适合需要快速适配新类别的少样本场景
Segment Anything Model（SAM）在基于提示的分割任务中表现出色，但语义文本提示的潜力远未被充分挖掘，相较于点、框等传统空间提示。本文提出SAM-PTx，一种基于冻结CLIP文本嵌入的参数高效微调方法，以类级别语义作为引导。具体而言，设计了一种名为Parallel-Text的轻量适配器，将文本嵌入注入SAM的图像编码器，实现语义引导的分割，同时保持大部分原始架构不变。该适配器仅修改每个Transformer块中的MLP并行分支，保留注意力路径用于空间推理。在COD10K数据集及COCO、ADE20K的低数据子集上的监督实验与消融分析表明，引入固定文本嵌入作为输入，可显著提升分割性能，优于纯空间提示基线。据我们所知，这是首个在COD10K数据集上使用文本提示进行分割的工作。结果表明，将语义条件融入SAM架构，为高效、可扩展的轻量级适配提供了可行路径。
原文摘要 · Abstract (English)
The Segment Anything Model (SAM) has demonstrated impressive generalization in prompt-based segmentation. Yet, the potential of semantic text prompts remains underexplored compared to traditional spatial prompts like points and boxes. This paper introduces SAM-PTx, a parameter-efficient approach for adapting SAM using frozen CLIP-derived text embeddings as class-level semantic guidance. Specifically, we propose a lightweight adapter design called Parallel-Text that injects text embeddings into SAM's image encoder, enabling semantics-guided segmentation while keeping most of the original architecture frozen. Our adapter modifies only the MLP-parallel branch of each transformer block, preserving the attention pathway for spatial reasoning. Through supervised experiments and ablations on the COD10K dataset as well as low-data subsets of COCO and ADE20K, we show that incorporating fixed text embeddings as input improves segmentation performance over purely spatial prompt baselines. To our knowledge, this is the first work to use text prompts for segmentation on the COD10K dataset. These results suggest that integrating semantic conditioning into SAM's architecture offers a practical and scalable path for efficient adaptation with minimal computational complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。