arXiv:2604.02816cs.CVcs.AI2026-04

提出量化感知的视觉标记剪枝方法,提升低比特多模态大模型推理精度。

QAPruner: Quantization-Aware Vision Token Pruning for Multimodal Large Language Models

  • 融合量化误差与异常值强度,动态评估视觉标记重要性。
  • 在仅保留12.5%视觉标记时,准确率比基线高2.24%,超越无剪枝量化。
  • 首次实现量化与剪枝协同优化,适合资源受限场景部署。

多模态大语言模型(MLLMs)虽具强大推理能力,但高昂的计算与内存开销限制了其在资源受限环境中的部署。尽管训练后量化(PTQ)和视觉标记剪枝是主流压缩技术,但通常被独立处理。本文发现二者强耦合:对已量化模型进行语义驱动剪枝可能丢弃对数值稳定性至关重要的激活异常值,导致低比特(如W4A4)下量化误差加剧。为此,我们提出量化感知视觉标记剪枝框架。该方法引入轻量级混合敏感度指标,结合模拟分组量化误差与异常值强度,再与标准语义相关性评分融合,保留兼具语义信息与量化鲁棒性的标记。在标准LLaVA架构上的实验表明,本方法持续优于朴素集成基线。在仅保留12.5%视觉标记的激进剪枝比下,准确率相较基线提升2.24%,甚至超越未剪枝的量化模型。据我们所知,这是首个显式协同优化视觉标记剪枝与PTQ的低比特MLLM推理方法。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) have shown strong reasoning ability, but their high computational and memory costs hinder deployment in resource-constrained settings. While Post-Training Quantization (PTQ) and vision token pruning are standard compression techniques, they are usually treated as independent optimizations. In this paper, we show that these two techniques are strongly coupled: naively applying semantic-based token pruning to PTQ-optimized MLLMs can discard activation outliers that are important for numerical stability and thus worsen quantization errors in low-bit regimes (\textit{e.g.}, W4A4). To address this issue, we propose a quantization-aware vision token pruning framework. Our method introduces a lightweight hybrid sensitivity metric that combines simulated group-wise quantization error with outlier intensity. By combining this metric with standard semantic relevance scores, the method retains tokens that are both semantically informative and robust to quantization. Experiments on standard LLaVA architectures show that our method consistently outperforms naive integration baselines. At an aggressive pruning ratio that retains only 12.5\% of visual tokens, our framework improves accuracy by 2.24\% over the baseline and even surpasses dense quantization without pruning. To the best of our knowledge, this is the first method that explicitly co-optimizes vision token pruning and PTQ for accurate low-bit MLLM inference.

多模态量化剪枝大模型压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。