arXiv:2512.12701cs.CVcs.CL2025-12中稿 · NeurIPS被引 1

动态剪枝视觉语言模型,40%算力节省下几乎不丢精度。

Efficient Vision-Language Reasoning via Adaptive Token Pruning

  • 按上下文重要性动态保留关键视觉与文本令牌
  • 推理时降低约40%计算量,速度提升1.5倍,精度损失<1%
  • 适合部署在资源受限的边缘多模态设备上

现实世界中视觉语言模型(VLMs)的部署受高计算需求限制,现有架构对所有令牌进行均匀处理。我们提出自适应令牌剪枝(ATP),一种在视觉-语言接口处动态保留最具信息量令牌的推理机制。ATP结合ViT CLS注意力(模态内显著性)和CLIP图文相似度(跨模态相关性)生成混合重要性评分,仅保留前K个令牌输入大语言模型。相比静态压缩,ATP可随输入自适应调整,无需修改骨干网络。作为轻量级门控模块,ATP兼容BLIP-2、LLaVA和Flamingo等主流模型。在VQAv2、GQA和COCO上的初步评估显示,ATP使推理FLOPs减少约40%,端到端延迟提速约1.5倍,精度损失小于1%。定性分析表明,ATP保持视觉定位能力并增强可解释性。此外,在噪声干扰下,自适应剪枝能抑制虚假关联,提升模型稳定性。这说明资源受限推理与模型可靠性并非对立目标。最后,我们探讨了ATP在高效多模态边缘计算流水线中的角色。

原文摘要 · Abstract (English)

Real-world deployment of Vision-Language Models (VLMs) is hindered by high computational demands, as existing architectures inefficiently process all tokens uniformly. We introduce Adaptive Token Pruning (ATP), a dynamic inference mechanism that retains only the most informative tokens based on contextual relevance. ATP operates at the vision-language interface, assigning a hybrid importance score combining ViT CLS attention (intra-modal saliency) and CLIP text-image similarity (inter-modal relevance) to keep top-K tokens for the LLM. Unlike static compression, ATP adapts to each input without modifying the backbone. Proposed as a lightweight gating module, ATP is compatible with popular backbones like BLIP-2, LLaVA, and Flamingo. Preliminary evaluations across VQAv2, GQA, and COCO indicate that ATP reduces inference FLOPs by around 40% and achieves roughly 1.5x speedups in end-to-end latency with negligible accuracy loss (less than 1%). Qualitative analyses suggest ATP preserves visual grounding and enhances interpretability. Beyond efficiency, we investigate robustness under corruptions; observations suggest adaptive pruning suppresses spurious correlations, improving stability. These findings imply that resource-constrained inference and model reliability are not competing objectives. Finally, we discuss ATP's role in efficient multimodal edge computing pipelines.

视觉语言模型动态剪枝边缘计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。