根据图像语义重要性动态分配压缩预算，提升视觉语言模型效率
Balancing Saliency and Coverage: Semantic Prominence-Aware Budgeting for Visual Token Compression in VLMs
- 按样本语义重要性分布自适应分配压缩预算
- 在88%算力降低下保持97.5%原始精度
- 适合需要高效视觉理解的部署场景
大型视觉语言模型（VLMs）通过高分辨率视觉输入实现强大的多模态理解能力，但由此产生的大量视觉标记带来了严重的计算瓶颈。近期工作通过视觉标记压缩缓解此问题，通常基于显著性、多样性或两者的固定组合进行压缩。我们观察到，不同样本间的语义重要性分布差异显著，导致局部显著性保留与全局覆盖之间的最优权衡各不相同。这表明对所有样本采用静态压缩策略可能次优。受此启发，我们提出 PromPrune：一种样本自适应的视觉标记选择框架，包含语义重要性感知的预算分配和两阶段选择流程。该方法根据每个样本的语义重要性分布，自适应平衡局部显著性保留与全局覆盖。通过在局部显著区域与全局多样区域之间分配标记预算，我们的方法在高压缩比下仍保持强性能。在 LLaVA-NeXT-7B 上，该方法将 FLOPs 降低 88%，预填充延迟减少 22%，同时保留 97.5% 的原始准确率。
原文摘要 · Abstract (English)
Large Vision-Language Models (VLMs) achieve strong multimodal understanding capabilities by leveraging high-resolution visual inputs, but the resulting large number of visual tokens creates a major computational bottleneck. Recent work mitigates this issue through visual token compression, typically compressing tokens based on saliency, diversity, or a fixed combination of both. We observe that the distribution of semantic prominence varies substantially across samples, leading to different optimal trade-offs between local saliency preservation and global coverage. This observation suggests that applying a static compression strategy across all samples can be suboptimal. Motivated by this insight, we propose PromPrune, a sample-adaptive visual token selection framework composed of semantic prominence-aware budget allocation and a two-stage selection pipeline. Our method adaptively balances local saliency preservation and global coverage according to the semantic prominence distribution of each sample. By allocating token budgets between locally salient regions and globally diverse regions, our method maintains strong performance even under high compression ratios. On LLaVA-NeXT-7B, our approach reduces FLOPs by 88% and prefill latency by 22% while preserving 97.5% of the original accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。