arXiv:2512.18747cs.CVcs.AI2025-12被引 8

提出无需训练的视觉编码压缩框架,高效减少多模态大模型计算量。

IPCV: Information-Preserving Compression for MLLM Visual Encoders

  • 通过邻域重建临时恢复被删视觉标记,参与注意力计算
  • 在多个图像视频基准上降低计算量并保持性能领先
  • 适合需要高效推理的多模态应用开发者

多模态大语言模型(MLLM)虽表现优异,但因视觉变换器(ViT)处理大量视觉标记导致计算开销高。现有剪枝策略存在缺陷:仅在语言模型阶段剪枝忽略ViT负担,而传统ViT剪枝缺乏语言引导,易丢弃关键视觉线索并引入特征畸变。为此,我们提出IPCV,一种无需训练的信息保留压缩框架。IPCV通过邻域引导重建(NGR)在ViT内部实现激进剪枝,临时重建被删标记以参与注意力计算,随后完全恢复再传给语言模型;同时引入注意力稳定化(AS)近似被剪枝标记的键值对,减轻剪枝负面影响。该方法可直接增强已有语言模型侧剪枝策略。大量实验表明,IPCV显著降低端到端计算量,在多种图像与视频基准上优于现有无训练剪枝方法。代码已公开于https://github.com/Perkzi/IPCV。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) deliver strong vision-language performance but at high computational cost, driven by numerous visual tokens processed by the Vision Transformer (ViT) encoder. Existing token pruning strategies are inadequate: LLM-stage token pruning overlooks the ViT's overhead, while conventional ViT token pruning, without language guidance, risks discarding textually critical visual cues and introduces feature distortions amplified by the ViT's bidirectional attention. To meet these challenges, we propose IPCV, a training-free, information-preserving compression framework for MLLM visual encoders. IPCV enables aggressive token pruning inside the ViT via Neighbor-Guided Reconstruction (NGR) that temporarily reconstructs pruned tokens to participate in attention with minimal overhead, then fully restores them before passing to the LLM. Besides, we introduce Attention Stabilization (AS) to further alleviate the negative influence from token pruning by approximating the K/V of pruned tokens. It can be directly applied to previous LLM-side token pruning methods to enhance their performance. Extensive experiments show that IPCV substantially reduces end-to-end computation and outperforms state-of-the-art training-free token compression methods across diverse image and video benchmarks. Our code is available at https://github.com/Perkzi/IPCV.

多模态模型视觉编码剪枝压缩高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。