arXiv:2604.02956cs.CV2026-04中稿 · ed被引 1

通过联合剪枝参数与视觉语言令牌,提升模型压缩效率。

Collaborative Multi-Mode Pruning for Vision-Language Models

论文配图:Collaborative Multi-Mode Pruning for Vision-Language Models
图 1 · 摘自论文原文
  • 设计协同重要性度量,动态评估参数与令牌的相互影响。
  • 在高剪枝率下仍保持性能,优于当前最先进方法。
  • 适合部署在资源受限设备上的多模态模型压缩。

视觉语言模型(VLMs)在统一的Transformer架构下快速发展,但其在资源受限设备上的部署仍面临计算复杂度高的挑战。虽然剪枝已成为压缩VLM的有效手段,但现有方法多仅针对单一模态(参数或令牌)进行剪枝,未能充分挖掘各模态内在冗余,导致高剪枝率下性能显著下降。为此,我们提出协同多模态剪枝(CoMP),一种专为VLM设计的联合参数与令牌剪枝框架。首先,构建协同重要性度量(CIM),分析耦合参数与令牌间的相互干扰,将令牌重要性融入参数重要性计算,同时缓解剪枝参数对令牌重要性评估的影响。其次,提出多模态剪枝策略(MPS),将整体剪枝过程分解为多个阶段,每阶段基于剪枝成本自适应选择最优剪枝模式,并融合历史成本与随机探索,确保剪枝过程稳定并避免局部最优。大量实验表明,该方法在多种视觉语言任务和模型上均显著优于现有最先进方法,在高剪枝率下仍保持优异性能。代码已开源:https://github.com/Wuzimeng/CoMP.git。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) have advanced rapidly within the unified Transformer architecture, yet their deployment on resource-constrained devices remains challenging due to high computational complexity. While pruning has emerged as an effective technique for compressing VLMs, existing approaches predominantly focus on a single mode by pruning either parameters or tokens, neglecting fully exploring the inherent redundancy in each mode, which leads to substantial performance degradation at high pruning ratios. To address the above limitations, we propose Collaborative Multi-Mode Pruning (CoMP), a novel framework tailored for VLMs by performing joint parameter and token pruning. Specifically, we first design a Collaborative Importance Metric (CIM) that investigates the mutual interference between the coupled parameters and tokens. It incorporates distinct significance of tokens into the computation of parameter importance scores, while simultaneously mitigating the affect of pruned parameters on token importance scores. Moreover, we develop a Multi-Mode Pruning Strategy (MPS) that decomposes the overall pruning process into a sequence of pruning stages, while in each stage we estimate the priory of different pruning modes based on their pruning cost and adaptively shift to the optimal one. Additionally, MPS integrates the historical cost and random exploration, in order to achieve a stable pruning process and avoid local optimum. Extensive experiments across various vision-language tasks and models demonstrate that our method effectively promotes the performance under high pruning ratios by comparing to the state-of-the-art approaches. The source code is available at https://github.com/Wuzimeng/CoMP.git.

模型压缩多模态剪枝

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。