arXiv:2512.21333cs.CV2025-12被引 3

通过文本引导剪枝,让SAM2视频分割更快更省显存。

Fast SAM2 with Text-Driven Token Pruning

  • 用轻量路由机制根据文本描述筛选关键视觉令牌
  • 推理速度提升42.50%,显存降低37.41%仍保持高精度
  • 适合实时或资源受限场景的高效视频分割应用

Segment Anything Model 2 (SAM2) 作为视觉基础模型,在提示驱动的视频对象分割上取得显著进展,但其实际部署受限于跨时间处理密集视觉令牌带来的高计算与内存开销。现有 SAM2 流水线通常将图像编码器生成的所有视觉令牌无差别传递至下游时序推理模块,导致因二次方复杂度的注意力内存开销而可扩展性下降。本文提出一种文本引导的令牌剪枝框架,在不修改底层分割架构的前提下,于视觉编码后、基于记忆的传播前,选择性降低令牌密度以提升推理效率。该方法利用轻量级路由机制,结合局部视觉上下文、基于目标中心文本描述(用户输入或自动生成)的语义相关性,以及不确定性线索,保留边界或模糊区域的关键令牌。实验表明,在多个挑战性视频分割基准上,后编码器剪枝实现高效且有效的提示感知分割,相比未剪枝的基线 SAM2,推理速度最高提升42.50%,GPU显存使用降低37.41%,同时保持具有竞争力的J and F性能。结果凸显了早期令牌选择对提升Transformer-based视频分割系统可扩展性的潜力。

原文摘要 · Abstract (English)

Segment Anything Model 2 (SAM2), a vision foundation model has significantly advanced in prompt-driven video object segmentation, yet their practical deployment remains limited by the high computational and memory cost of processing dense visual tokens across time. The SAM2 pipelines typically propagate all visual tokens produced by the image encoder through downstream temporal reasoning modules, regardless of their relevance to the target object, resulting in reduced scalability due to quadratic memory attention overhead. In this work, we introduce a text-guided token pruning framework that improves inference efficiency by selectively reducing token density prior to temporal propagation, without modifying the underlying segmentation architecture. Operating after visual encoding and before memory based propagation, our method ranks tokens using a lightweight routing mechanism that integrates local visual context, semantic relevance derived from object-centric textual descriptions (either user-provided or automatically generated), and uncertainty cues that help preserve ambiguous or boundary critical regions. By retaining only the most informative tokens for downstream processing, the proposed approach reduces redundant computation while maintaining segmentation fidelity. Extensive experiments across multiple challenging video segmentation benchmarks demonstrate that post-encoder token pruning provides a practical and effective pathway to efficient, prompt-aware video segmentation, achieving up to 42.50 percent faster inference and 37.41 percent lower GPU memory usage compared to the unpruned baseline SAM2, while preserving competitive J and F performance. These results highlight the potential of early token selection to improve the scalability of transformer-based video segmentation systems for real-time and resource-constrained applications.

视频分割SAM2高效推理文本引导

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。