只对关键区域高分辨率处理，提升视觉语言模型效率与精度
Look Where It Matters: High-Resolution Crops Retrieval for Efficient VLMs
- 按需提取图像局部高分辨率片段，避免全图高分辨率计算
- 在多个数据集上实现精度不降、推理时间减少30%以上
- 适合需要高效精准图文理解的场景，如文档分析、细粒度识别
视觉语言模型通常以原生高分辨率处理图像，导致精度与效率间的权衡：高分辨率能捕捉细节但计算成本高，低分辨率虽高效却可能遗漏关键信息（如小文本）。我们提出AwaRes，一种按需空间框架，先以低分辨率全局视图进行处理，再通过工具调用检索查询所需的高分辨率片段。我们自动生成监督数据：由裁判比较低/高分辨率答案，判断是否需要裁剪；由接地模型定位正确答案的证据，并映射为离散裁剪集合，形成多轮工具使用轨迹。采用冷启动SFT后接多轮GRPO训练，复合奖励包含语义答案正确性与显式裁剪成本惩罚。项目主页：https://nimrodshabtay.github.io/AwaRes
原文摘要 · Abstract (English)
Vision-language models (VLMs) typically process images at a native high-resolution, forcing a trade-off between accuracy and computational efficiency: high-resolution inputs capture fine details but incur significant computational costs, while low-resolution inputs advocate for efficiency, they potentially miss critical visual information, like small text. We present AwaRes, a spatial-on-demand framework that resolves this accuracy-efficiency trade-off by operating on a low-resolution global view and using tool-calling to retrieve only high-resolution segments needed for a given query. We construct supervised data automatically: a judge compares low- vs.\ high-resolution answers to label whether cropping is needed, and an oracle grounding model localizes the evidence for the correct answer, which we map to a discrete crop set to form multi-turn tool-use trajectories. We train our framework with cold-start SFT followed by multi-turn GRPO with a composite reward that combines semantic answer correctness with explicit crop-cost penalties. Project page: https://nimrodshabtay.github.io/AwaRes
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。