让视觉语言模型学会在需要时回看局部细节,提升细节理解能力。
Learning to Look Again: Loss-Gap Supervision for Free-form Crop Routing in Vision-Language Models

- 基于目标模型自身错误信号,自动学习何时何地回看局部图像。
- 在6个基准上平均提升12.04分,最高达64.29,超越现有方法。
- 轻量级路由模块可适配不同任务,精准控制计算开销。
视觉语言模型(VLMs)在回答依赖细节的问题时表现不佳,原因在于图像被压缩为低分辨率全局视图后,关键信息丢失。虽然增加视觉标记可改善部分文档和OCR任务,但会浪费计算资源并干扰依赖全局上下文的任务。本文提出GapSight框架,实现视觉重读:模型先进行全局扫描,当问题需要局部证据时,再选择性返回自由形式区域。监督信号来自目标模型自身的失败信号——通过对比仅全局视图与候选裁剪视图下的答案损失或选项置信度差距,筛选出能提升答案的裁剪区域作为模型专属的重读标签。一个轻量级自由裁剪路由器将这些标签蒸馏为一次性推理策略,预测是否重读、预期收益及连续裁剪框。在LLaVA-1.5-7B、InternVL2.5-8B和Qwen2-VL-2B-Instruct上,GapSight在六个涵盖OCR、文档、图表、信息图、VStarBench和MME-RealWorld-Lite的基准上均优于无缩放基线。在InternVL2.5-8B上,六项基准平均得分从52.25提升至64.29,超过CropVLM(57.16)、ViCrop(55.84)和ZoomRefine(54.43)。机制分析表明,路由器有效纠正具体错误答案,按任务自适应动作频率,并形成更优的令牌-性能分布。
原文摘要 · Abstract (English)
Vision-language models (VLMs) fail many detail-centric questions for a concrete reason: the answer is visible in the image, yet lost after the image is compressed into a low-resolution global view. Allocating more visual tokens to every query improves some OCR and document cases, but it spends computation indiscriminately and can disturb tasks that rely on global context. We propose GapSight, a framework for learning visual re-reading: a VLM first takes a global glance, then selectively returns to a free-form region when the question calls for local evidence. The supervision comes from the target model's own failure signal. Offline, we compare answer loss or multiple-choice option margin under a global-only view and candidate crop-augmented views; crops that improve the target answer become model-specific review labels. A lightweight free-form crop router distills these labels into a one-shot inference policy that predicts whether to review, expected utility, and a continuous crop box from the global state. Across LLaVA-1.5-7B, InternVL2.5-8B, and Qwen2-VL-2B-Instruct, GapSight improves the Base no-zoom baseline on six benchmarks spanning OCR, documents, charts, infographics, VStarBench, and MME-RealWorld-Lite. On InternVL2.5-8B, GapSight raises the six-benchmark average from 52.25 to 64.29, above CropVLM (57.16), ViCrop (55.84), and ZoomRefine (54.43). Mechanism analyses show that the router rescues concrete wrong answers, adapts its action rate by task, and forms a favorable token-performance profile. These results position loss-gap supervision as a practical route to teaching VLMs when and where to look again.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。