arXiv:2509.21991cs.CVcs.AI2025-09被引 1

通过分阶段聚焦关键图像区域,显著降低视觉语言模型的计算开销。

ERGO: Efficient High-Resolution Visual Understanding for Vision-Language Models

  • 先粗后精:先低分辨率分析定位重点区域,再局部高精度处理。
  • 在V*基准上比Qwen2.5-VL-7B高4.7分,仅用23%视觉标记,推理快3倍。
  • 基于强化学习动态扩展模糊区域,提升复杂问题的准确率,适合实际部署。

高效处理高分辨率图像对真实世界的视觉语言应用至关重要。现有大型视觉语言模型(LVLMs)因视觉标记数量庞大而带来巨大计算负担。随着“以图像思考”模型的出现,推理已从文本延伸至视觉领域。为此,我们提出两阶段“由粗到精”的推理流程:首先对下采样图像进行分析,识别与任务相关区域;随后仅对这些区域以全分辨率裁剪并处理。该方法在保持细粒度视觉细节的同时显著降低计算成本。主要挑战在于如何准确判断哪些区域真正与查询相关。现有方法在图像下采样后常因依赖感知推理而失效,因需清晰视觉信息才能有效推理。为此,我们提出ERGO(Efficient Reasoning & Guided Observation),利用多模态上下文驱动的推理来确定关注点。模型可处理感知不确定性,自动扩展裁剪区域以覆盖视觉模糊区域。为此,我们在强化学习框架中设计了简单有效的奖励机制。在多个数据集上,我们的方法在准确率上优于原模型及竞争方法,且效率更高。例如,在V*基准上,ERGO比Qwen2.5-VL-7B高出4.7分,仅使用23%的视觉标记,实现3倍推理加速。

原文摘要 · Abstract (English)

Efficient processing of high-resolution images is crucial for real-world vision-language applications. However, existing Large Vision-Language Models (LVLMs) incur substantial computational overhead due to the large number of vision tokens. With the advent of "thinking with images" models, reasoning now extends beyond text to the visual domain. This capability motivates our two-stage "coarse-to-fine" reasoning pipeline: first, a downsampled image is analyzed to identify task-relevant regions; then, only these regions are cropped at full resolution and processed in a subsequent reasoning stage. This approach reduces computational cost while preserving fine-grained visual details where necessary. A major challenge lies in inferring which regions are truly relevant to a given query. Recent related methods often fail in the first stage after input-image downsampling, due to perception-driven reasoning, where clear visual information is required for effective reasoning. To address this issue, we propose ERGO (Efficient Reasoning & Guided Observation) that performs reasoning-driven perception-leveraging multimodal context to determine where to focus. Our model can account for perceptual uncertainty, expanding the cropped region to cover visually ambiguous areas for answering questions. To this end, we develop simple yet effective reward components in a reinforcement learning framework for coarse-to-fine perception. Across multiple datasets, our approach delivers higher accuracy than the original model and competitive methods, with greater efficiency. For instance, ERGO surpasses Qwen2.5-VL-7B on the V* benchmark by 4.7 points while using only 23% of the vision tokens, achieving a 3x inference speedup. The code and models can be found at: https://github.com/nota-github/ERGO.

视觉语言模型高效推理图像理解分阶段处理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。