让视觉语言模型自动聚焦细节区域，提升细粒度理解能力。
CropVLM: Learning to Zoom for Fine-Grained Vision-Language Perception
- 通过强化学习训练模型动态选择图像关键区域
- 在无标注框情况下显著提升细粒度任务表现
- 无需修改原模型，适合各类开源与商用视觉语言模型
视觉语言模型在需要细粒度图像理解的任务（如场景文本识别、文档分析）中表现受限，主要由于感知能力不足和视觉碎片化。为此，我们提出CropVLM，一种低成本外部增强方法，使模型能动态“放大”关注图像中的相关区域，从而增强对细节的捕捉能力。CropVLM采用强化学习进行训练，不依赖人工标注的边界框作为监督信号，也无需昂贵的合成评估。该模型只需训练一次，即可与开源及专有视觉语言模型配合使用，显著提升其性能。该方法在需高分辨率理解的任务上表现突出，尤其在目标模型未见过的领域数据上效果显著，且无需修改或微调原始模型，避免灾难性遗忘。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) often struggle with tasks that require fine-grained image understanding, such as scene-text recognition or document analysis, due to perception limitations and visual fragmentation. To address these challenges, we introduce CropVLM as an external low-cost method for boosting performance, enabling VLMs to dynamically ''zoom in'' on relevant image regions, enhancing their ability to capture fine details. CropVLM is trained using reinforcement learning, without using human-labeled bounding boxes as a supervision signal, and without expensive synthetic evaluations. The model is trained once and can be paired with both open-source and proprietary VLMs to improve their performance. Our approach delivers significant improvements on tasks that require high-resolution image understanding, notably for benchmarks that are out-of-domain for the target VLM, without modifying or fine-tuning the VLM, thus avoiding catastrophic forgetting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。