arXiv:2604.11240cs.CV2026-04中稿 · ACM Multimedia 202…

提出新方法,让大模型更智能地删减图像信息,提速降耗不丢精度。

Decoupled Similarity for Task-Aware Token Pruning in Large Vision-Language Models

  • 用跨模态相似度解耦机制,精准判断哪些图像区域该保留
  • 在保留11.1%图像标记时,性能损失仅1.9%,推理速度提升2.3倍
  • 适合追求高效视觉语言模型部署的研究者和工程师

令牌剪枝已成为降低大型视觉语言模型(LVLMs)计算开销的有效方法,通过剔除低信息量的视觉令牌来实现。然而,现有方法通常依赖于不同组件的独立注意力源,导致剪枝决策不完整且次优,因注意力分布存在偏差。为此,我们提出一种新的解耦相似度感知剪枝方法(DeSAP),用于视觉编码器中的精确、任务感知剪枝。具体而言,DeSAP引入解耦相似度,捕捉视觉特征与文本令牌之间的细粒度跨模态相关性,为剪枝提供明确的任务导向指引。结合从视觉注意力中提取的视觉显著性信号,DeSAP在任务相关性和视觉线索双重指导下进行剪枝,即使在激进剪枝比例下仍保持鲁棒性。在多种基准和架构上的广泛实验表明,DeSAP在准确率和效率方面均持续优于当前最优方法。在LLaVA-1.5-7B上,仅保留11.1%的视觉令牌即可实现10倍的浮点运算量减少和2.3倍的预填充加速,同时保持98.1%的原始性能。

原文摘要 · Abstract (English)

Token pruning has emerged as an effective approach to reduce the substantial computational overhead of Large Vision-Language Models (LVLMs) by discarding less informative visual tokens while preserving performance. However, existing methods typically rely on individual attention sources from different LVLM components, resulting in incomplete and suboptimal pruning decisions due to biased attention distributions. To address this problem, we propose DeSAP, a novel Decoupled Similarity-Aware Pruning method for precise, task-aware token pruning within the visual encoder. Specifically, DeSAP introduces a decoupled similarity to capture fine-grained cross-modal relevance between visual features and text tokens, providing explicit task-related guidance for pruning. By integrating decoupled similarity with visual saliency signals derived from visual attention, DeSAP performs token pruning under the guidance of both task-related and visual cues, enabling robust pruning even under aggressive pruning ratios. Extensive experiments across diverse benchmarks and architectures show that DeSAP consistently outperforms SOTA methods in both accuracy and efficiency. On LLaVA-1.5-7B, DeSAP achieves a 10 times FLOPs reduction and a 2.3 times prefill speedup by retaining only 11.1% of visual tokens, while maintaining 98.1% of the original performance.

视觉语言模型令牌剪枝高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。