用语义超像素压缩视觉令牌,93%降冗余仍保精度
Re-purposing SAM into Efficient Visual Projectors for MLLM-Based Referring Image Segmentation
- 以SAM生成的语义超像素替代传统图像块,作为视觉词
- 视觉令牌减少93%且不丢性能,训练推理速度显著提升
- 适合追求高效多模态模型部署的研究者与工程师
近期将多模态大语言模型(MLLM)与分割一切模型(SAM)结合的指代图像分割(RIS)框架取得了显著进展。然而,将MLLM适配到分割任务计算开销大,主要源于视觉令牌冗余。我们发现传统基于补丁的视觉投影器难以在减少视觉令牌数量和保持语义清晰度之间取得平衡,常需保留过长的令牌序列以避免性能下降。受文本分词启发,我们提出一种新型语义视觉投影器,利用SAM生成的语义超像素识别图像中的“视觉词”。通过将语义超像素压缩并投影为视觉令牌,该方法能根据场景复杂度自适应缩短令牌序列,同时最小化压缩带来的语义损失。为缓解信息丢失,我们设计了语义超像素位置嵌入,增强MLLM对超像素几何与位置的感知,并引入语义超像素聚合器,以保留超像素内部的细粒度细节和外部全局上下文。实验表明,该方法在不牺牲性能的前提下,将视觉令牌减少93%,显著加速了MLLM的训练与推理,优于现有压缩型视觉投影器在RIS任务上的表现。
原文摘要 · Abstract (English)
Recently, Referring Image Segmentation (RIS) frameworks that pair the Multimodal Large Language Model (MLLM) with the Segment Anything Model (SAM) have achieved impressive results. However, adapting MLLM to segmentation is computationally intensive, primarily due to visual token redundancy. We observe that traditional patch-wise visual projectors struggle to strike a balance between reducing the number of visual tokens and preserving semantic clarity, often retaining overly long token sequences to avoid performance drops. Inspired by text tokenizers, we propose a novel semantic visual projector that leverages semantic superpixels generated by SAM to identify "visual words" in an image. By compressing and projecting semantic superpixels as visual tokens, our approach adaptively shortens the token sequence according to scene complexity while minimizing semantic loss in compression. To mitigate loss of information, we propose a semantic superpixel positional embedding to strengthen MLLM's awareness of superpixel geometry and position, alongside a semantic superpixel aggregator to preserve both fine-grained details inside superpixels and global context outside. Experiments show that our method cuts visual tokens by 93% without compromising performance, notably speeding up MLLM training and inference, and outperforming existing compressive visual projectors on RIS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。