通过分层筛选关键视觉区域和令牌，显著降低高分辨率多模态模型的计算开销。
Pyramid Token Pruning for High-Resolution Large Vision-Language Models via Region, Token, and Instruction-Guided Importance
- 结合视觉显著性与任务指令，分层筛选重要视觉区域和令牌
- 在13个基准上减少计算量、内存占用和推理延迟，性能几乎不变
- 无需训练，适合部署高分辨率多模态模型的场景
大型视觉语言模型（LVLMs）近期展现出强大的多模态理解能力，但其细粒度视觉感知常受限于低输入分辨率。常见方法是将高分辨率图像划分为多个子图像分别编码，但这大幅增加视觉令牌数量，导致严重的推理开销。为此，我们提出金字塔令牌剪枝（PTP），一种无需训练的策略，通过自底向上融合区域与令牌级别的视觉显著性，以及自顶向下任务指令引导的相关性，实现层次化筛选。受人类视觉认知启发，PTP优先保留显著区域的更多令牌，并进一步强调与任务指令最相关的内容。在13个多样化基准上的大量实验表明，PTP显著降低计算成本、内存使用和推理延迟，性能损失可忽略不计。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) have recently demonstrated strong multimodal understanding, yet their fine-grained visual perception is often constrained by low input resolutions. A common remedy is to partition high-resolution images into multiple sub-images for separate encoding, but this approach drastically inflates the number of visual tokens and introduces prohibitive inference overhead. To overcome this challenge, we propose Pyramid Token Pruning (PTP), a training-free strategy that hierarchically integrates bottom-up visual saliency at both region and token levels with top-down instruction-guided relevance. Inspired by human visual cognition, PTP selectively preserves more tokens from salient regions while further emphasizing those most relevant to task instructions. Extensive experiments on 13 diverse benchmarks show that PTP substantially reduces computational cost, memory usage, and inference latency, with negligible performance degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。