针对视觉语言模型的异构特性，提出不对称剪枝以高效压缩推理。
AsymVLM: Asymmetric Token Pruning for Efficient Vision-Language Model Inference
- 对视觉令牌做激进剪枝，文本令牌按时间阈值剔除，适应模态差异
- 最高节省54%计算量，在图表理解任务上提升2-3%准确率
- 适合视觉信息局部化、查询特定的场景，如文档分析
视觉语言模型每张图像处理数千个视觉标记，而文本标记数量较少，但现有压缩方法对两种模态一视同仁。我们观察到两者本质不同：视觉标记具有空间冗余性，主导预填充阶段；文本标记具有因果依赖性，在解码中累积。基于此不对称性，我们提出并实证评估了AsymVLM，该方法在预填充前使用学习的重要性评分器对视觉标记进行激进剪枝，并采用基于时间阈值的剔除策略仅在文本标记超过固定预算时触发。实验表明，AsymVLM在现有方法中实现最高FLOPs节省（最高达54%），在文档与图表理解任务中表现优于现有方法2–3%，这些任务中视觉信息空间局部且查询特定；在整体基准上保持竞争力。在文本主导场景下，其剔除策略显著优于标准LLM缓存压缩方法，因更适配视觉语言模型短上下文特性。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) process thousands of visual tokens per image alongside comparatively few text tokens, yet existing compression methods treat both modalities uniformly. We observe that the two modalities have fundamentally different properties: vision tokens are spatially redundant and dominate prefill, while text tokens are causally dependent and accumulate during decoding. Based on this asymmetry, we propose and empirically evaluate AsymVLM, which applies aggressive pruning to vision tokens before prefill using a learned importance scorer with per-sample adaptive budgeting, and temporal threshold-based eviction to text tokens only when they exceed a fixed budget. Our experiments indicate that AsymVLM achieves the highest FLOPs savings (up to 54%) among state-of-the-art methods while outperforming existing approaches by 2--3% on document and chart understanding tasks where visual information is spatially localized and query-specific, and maintaining competitive accuracy on holistic benchmarks. In text-dominated scenarios, our eviction strategy substantially outperforms standard LLM cache compression methods by adapting to the short-context nature of VLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。