轻量化文本编码器让视觉语言分割更高效
SAM3-LiteText: An Anatomical Study of the SAM3 Text Encoder for Efficient Vision-Language Segmentation
- 用知识蒸馏训练轻量MobileCLIP替代原文本编码器
- 参数减少88%，静态内存大幅降低，性能基本不变
- 适合需要低延迟、低资源的视觉分割应用
视觉语言分割模型如SAM3可实现灵活的提示驱动视觉定位，但依赖于为开放式语言理解设计的大规模通用文本编码器。实际分割提示短小、结构化且语义受限，导致文本编码器能力严重过剩，持续带来计算与内存开销。本文对视觉语言分割中的文本提示进行大规模解剖分析，涵盖404,796个真实提示。结果表明：多数上下文窗口利用率低，词汇使用高度稀疏，文本嵌入虽在高维空间却位于低维流形。基于此，我们提出SAM3-LiteText，用通过知识蒸馏优化的轻量MobileCLIP学生模型替代原编码器。在图像与视频分割基准上实验显示，该方法将文本编码器参数减少高达88%，显著降低静态内存占用，同时保持与原模型相当的分割性能。
原文摘要 · Abstract (English)
Vision-language segmentation models such as SAM3 enable flexible, prompt-driven visual grounding, but inherit large, general-purpose text encoders originally designed for open-ended language understanding. In practice, segmentation prompts are short, structured, and semantically constrained, leading to substantial over-provisioning in text encoder capacity and persistent computational and memory overhead. In this paper, we perform a large-scale anatomical analysis of text prompting in vision-language segmentation, covering 404,796 real prompts across multiple benchmarks. Our analysis reveals severe redundancy: most context windows are underutilized, vocabulary usage is highly sparse, and text embeddings lie on low-dimensional manifold despite high-dimensional representations. Motivated by these findings, we propose SAM3-LiteText, a lightweight text encoding framework that replaces the original SAM3 text encoder with a compact MobileCLIP student that is optimized by knowledge distillation. Extensive experiments on image and video segmentation benchmarks show that SAM3-LiteText reduces text encoder parameters by up to 88%, substantially reducing static memory footprint, while maintaining segmentation performance comparable to the original model. Code: https://github.com/SimonZeng7108/efficientsam3/tree/sam3_litetext.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。