提出分步精简视觉标记的新方法，提升视觉语言模型效率与准确性。
Focus-then-Context: Subject-Centric Progressive Visual Token Reduction for Vision-Language Models
- 先聚焦主体再捕捉上下文，模仿人类视觉认知过程
- 仅保留22.2%的视觉标记，速度提升2.53倍，准确率损失仅0.6%
- 适合需要高效推理的多模态应用，如智能客服、图像理解
视觉语言模型在推理时因海量视觉标记序列面临计算成本过高的瓶颈。现有标记压缩方法虽减轻负担，却过度保留与查询严格对齐的孤立主体，难以探索显著主体及其上下文关系。本文提出SPpruner，一种以主体为中心的渐进式压缩范式，模拟人类视觉系统的‘聚焦-上下文’机制。首先构建焦点识别模块，显式建模视觉显著性与语义相关性的交互，挖掘完整的主体谱系以保证输入表征的高保真度；随后设计上下文感知结构扫描模块，聚合邻域上下文线索，有效恢复全局关系依赖，维持保留主体的结构完整性。大量实验表明，该范式持续优于当前最优方法：在Qwen2.5-VL上实现最高2.53倍加速，仅需22.2%视觉标记；在LLaVA上减少67%浮点运算量，精度仅下降0.6%。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) face a bottleneck of prohibitive computational costs arising from massive visual token sequences during inference. Existing vision token reduction methods alleviate this burden, but they unintentionally preserve the isolated visual subject strictly aligned with the user's query, which fails to substantially explore salient subjects and their contextual relationships. In this paper, we propose SPpruner, a subject-centric progressive reduction paradigm that emulates the \textit{Focus-then-Context} mechanism of the human visual perception system. Specifically, we first construct a focus identification module to explicitly model the interplay between visual saliency and semantic relevance. Herein, it can excavate the comprehensive visual subject spectrum to ensure a high-fidelity representation of visual input. Subsequently, a context-aware structural scanning module is developed to aggregate contextual cues from neighboring regions. As such, it can effectively restore global relational dependencies to uphold the structural integrity of the preserved subjects. Extensive experiments demonstrate that our paradigm consistently outperforms SOTA methods, achieving up to 2.53 times speedup with only 22.2% of visual tokens retained in Qwen2.5-VL and a 67% FLOPs reduction on LLaVA with a negligible 0.6% accuracy drop.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。