不训练即可高效剪枝视觉语言模型中的冗余图像标记
ZOO-Prune: Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Models
- 用零阶梯度估计计算每个图像标记对输出的影响程度
- 剪掉高达94.4%的标记，精度几乎不变，推理速度提升2.3倍
- 无需训练，适用于多种视觉语言模型，适合部署优化
大型视觉语言模型（VLMs）虽具备强大的多模态推理能力，但冗余的视觉标记带来了高昂的推理开销。现有剪枝方法存在局限：基于注意力的方法依赖原始注意力分数，层与头间不稳定，易选冗余标记；基于多样性的方法虽更鲁棒，但可能误删关键区域。本文提出ZOO-Prune，一种无需训练的剪枝框架，其核心思想是：对输出敏感的标记影响力更强，且捕捉互补视觉线索而非冗余信息。通过在轻量投影层使用零阶扰动估计标记敏感性，无需反向传播即可高效近似每个标记的影响。在多个VLM和基准测试上的实验表明，ZOO-Prune持续优于现有方法，最多可剪除94.4%的标记而保持精度，并使端到端推理速度最高提升2.30倍。
原文摘要 · Abstract (English)
Large Vision-Language Models (VLMs) enable strong multimodal reasoning but incur heavy inference costs from redundant visual tokens. Token pruning alleviates this issue, yet existing approaches face limitations. Attention-based methods rely on raw attention scores, which are often unstable across layers and heads and can lead to redundant selections. Diversity-based methods improve robustness by selecting tokens far apart in feature space, but risk dropping regions needed for accurate prediction. We propose ZOO-Prune, a training-free framework built on the intuition that highly sensitive tokens have a stronger influence on the model's output and capture complementary visual cues rather than redundant ones. To achieve this, we estimate token sensitivity using zeroth-order perturbations at the lightweight projection layer. This measures how small random perturbations affect the projected features and enables efficient approximation of each token's influence without backpropagation. Extensive experiments across multiple VLMs and benchmarks show that ZOO-Prune consistently outperforms prior methods while pruning up to 94.4% of tokens without sacrificing accuracy. Our method also improves efficiency, reaching up to 2.30x faster end-to-end inference compared to the baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。