arXiv:2603.16001cs.CVcs.CL2026-03

针对图文模型剪枝,提出不对称文本视觉剪枝方法。

Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Models

  • 分别用文本和视觉令牌校准,提升剪枝精度
  • 视觉路径可高达50%稀疏度,文本路径更敏感需精细处理
  • 适合追求高效推理的多模态模型部署场景

网络剪枝是实现轻量化大型视觉语言模型(LVLMs)的有效手段,通常将权重和激活值纳入重要性评估。然而,现有方法对不同模态的校准数据采用统一处理方式,忽视了模态特异性行为。这带来关键挑战:如何准确捕捉文本与视觉标记在剪枝中的差异行为。我们通过解耦对应权重,系统研究了两类标记对剪枝的敏感性,发现:(i) 文本路径应基于文本标记校准,因其敏感性高于视觉路径;(ii) 视觉路径具有高度冗余性,允许高达50%的稀疏度。基于此,我们提出简单高效的非对称文本-视觉权重剪枝方法ATV-Pruning,通过从文本和视觉路径中选取信息量高的标记构建重要性度量。具体创新包括:(1) 自适应构建校准池,融合全部文本标记与部分视觉标记;(2) 设计层自适应选择策略筛选重要视觉标记。大量实验在标准多模态基准上验证了ATV-Pruning优于当前最先进方法。

原文摘要 · Abstract (English)

Network pruning is an effective technique for enabling lightweight Large Vision-Language Models (LVLMs), which primarily incorporates both weights and activations into the importance metric. However, existing efforts typically process calibration data from different modalities in a unified manner, overlooking modality-specific behaviors. This raises a critical challenge: how to address the divergent behaviors of textual and visual tokens for accurate pruning of LVLMs. To this end, we systematically investigate the sensitivity of visual and textual tokens to the pruning operation by decoupling their corresponding weights, revealing that: (i) the textual pathway should be calibrated via text tokens, since it exhibits higher sensitivity than the visual pathway; (ii) the visual pathway exhibits high redundancy, permitting even 50% sparsity. Motivated by these insights, we propose a simple yet effective Asymmetric Text-Visual Weight Pruning method for LVLMs, dubbed ATV-Pruning, which establishes the importance metric for accurate weight pruning by selecting the informative tokens from both textual and visual pathways. Specifically, ATV-Pruning integrates two primary innovations: first, a calibration pool is adaptively constructed by drawing on all textual tokens and a subset of visual tokens; second, we devise a layer-adaptive selection strategy to yield important visual tokens. Finally, extensive experiments across standard multimodal benchmarks verify the superiority of our ATV-Pruning over state-of-the-art methods.

模型剪枝视觉语言模型轻量化多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。