无需预训练，用对比学习让大模型学会高效缩放。
Learning to Zoom Efficiently with a Contrastive Curriculum
- 设计渐进式难负样本的对比奖励机制，驱动模型自适应学习缩放。
- 在V*、HRBench等数据集上性能优于有监督微调，且更高效。
- 提出新数据集M&C，发现召回率最能反映缩放能力与任务表现的关系。
使用缩放工具是现代视觉智能体的关键能力，可有效处理高分辨率图像任务。以往方法需大量有监督微调来教会模型缩放，我们证明这并非必需。本文提出一种基于InfoNCE的内在奖励机制，通过逐步增加难度的负向工具调用作为对比信号，在无需额外标签或预热微调的情况下训练多模态大模型（MLLM）掌握缩放技能。在V*、HRBench和MME-RealWorld上的实验证明，该方法性能具有竞争力且效率更高；作为现有SFT的直接替代，甚至超越所有基线。为准确评估模型缩放能力，我们引入可扩展的合成数据集Muffin&Chihuahua（M&C），每张图由格子构成，每个格子为蛋糕或吉娃娃。利用其精确的感兴趣区域标注，我们发现召回率与最终任务性能相关性最强。模型与代码已开源：https://github.com/UKPLab/emnlp2026-zoom-in。
原文摘要 · Abstract (English)
Using a zoom-in tool is an important foundational part of modern visual agents, because it allows to efficiently handle tasks involving high-resolution images. Most previous methods need an extensive warm-start supervised fine-tuning phase for teaching models zoom-in. We show that this is not necessary by proposing a new intrinsic reward for learning tool use in MLLMs without the need for additional labels or warm-start SFT. Our InfoNCE-style reward uses a curriculum of increasingly hard negative tool calls as a contrastive training signal. Empirical experiments on $V^*$, HRBench and MME-RealWorld show that our approach is competitive while being more efficient. When used as a drop-in replacement for SFT, we even outperform all baselines. To directly measure the zoom-in ability of models, we further introduce the scalable synthetic Muffin&Chihuahua (M&C) dataset. Each image consists of a grid with every cell either showing a muffin or chihuahua. Leveraging the M&C dataset's unique region of interest labels, we find that recall is the metric that most strongly correlates the zoom-in region with final task performance. Our model and code for reproduction is publicly available under https://github.com/UKPLab/emnlp2026-zoom-in
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。