arXiv:2509.13067cs.CV2025-09被引 2

提出高效方法,按重要性删减高分辨率图像的视觉标记,提升大模型推理速度。

HERO: Rethinking Visual Token Early Dropping in High-Resolution Large Vision-Language Models

  • 根据视觉重要性和任务相关性动态评估图像块重要性,决定保留哪些标记。
  • 在多个基准上实现更高效率与精度平衡,且无需重新训练模型。
  • 适合追求高效推理的高分辨率视觉语言模型研究与应用者。

将高分辨率图像切分为局部块并独立编码,使高分辨率大型视觉语言模型(HR-LVLMs)展现出出色的细粒度视觉理解能力。然而,这种分而治之范式显著增加了视觉标记数量,带来巨大的计算与内存开销。我们通过实证研究视觉标记利用率,发现三个关键现象:(1) 局部块的重要性由视觉显著性与任务相关性共同决定;(2) 基于CLIP的视觉编码器中CLS token在各层呈现双阶段注意力模式,每阶段关注不同类型的视觉标记;(3) 不同阶段强调的视觉标记编码不同粒度的信息,在模型中发挥互补作用。基于此,我们提出HERO——一种高分辨率视觉标记早期丢弃框架,结合内容自适应标记预算分配与功能感知标记选择。通过精准估计块级重要性并保留具有互补作用的视觉标记,HERO在多种基准和模型规模下均实现更优的效率-精度权衡,且全程无需训练。本研究为高分辨率视觉语言模型高效推理提供了实证洞察与实用方案。

原文摘要 · Abstract (English)

By cropping high-resolution images into local tiles and encoding them independently, High-Resolution Large Vision-Language Models (HR-LVLMs) have demonstrated remarkable fine-grained visual understanding capabilities. However, this divide-and-conquer paradigm significantly increases the number of visual tokens, resulting in substantial computational and memory overhead. To better understand and address this challenge, we empirically investigate visual token utilization in HR-LVLMs and uncover three key findings: (1) the local tiles have varying importance, jointly determined by visual saliency and task relevance; (2) the CLS token in CLIP-based vision encoders exhibits a two-stage attention pattern across layers, with each stage attending to different types of visual tokens; (3) the visual tokens emphasized at different stages encode information at varying levels of granularity, playing complementary roles within LVLMs. Building on these insights, we propose HERO, a High-resolution visual token early dropping framework that integrates content-adaptive token budget allocation with function-aware token selection. By accurately estimating tile-level importance and selectively retaining visual tokens with complementary roles, HERO achieves superior efficiency-accuracy trade-offs across diverse benchmarks and model scales, all in a training-free manner. This study provides both empirical insights and practical solutions toward efficient inference in HR-LVLMs.

视觉语言模型高效推理图像分割

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。