让大模型通过多轮对话自动聚焦图像关键区域,提升高分辨率视觉推理能力。
High-Resolution Visual Reasoning via Multi-Turn Grounding-Based Reinforcement Learning
- 基于多轮对话设计自适应裁剪策略,引导模型逐步聚焦关键视觉区域。
- 仅用答案正确性作为奖励,训练后在真实场景和分布外数据上分别提升5.4%和5.2%。
- 无需额外标注,适合希望提升视觉定位能力的模型优化者使用。
当前主流大模型在处理高分辨率图像时面临挑战,因输入生成大量无关视觉标记。本文提出多轮锚定式策略优化(MGPO),一种端到端强化学习框架,使大模型在多轮对话中根据预测的锚定坐标自动裁剪子图像,迭代聚焦关键视觉区域。与需额外标注的监督微调相比,本方法表明大模型可在强化学习过程中自发形成鲁棒的视觉定位能力,仅依赖最终答案正确性的二元奖励信号。此外,我们发现大模型在推理阶段难以自主触发视觉锚定,为此设计多轮对话模板,并将策略损失计算限制在多轮对话输出上,以促进稳定优化。大量实验表明,仅在标准视觉问答数据上训练、无标注的情况下,MGPO相比GRPO在分布内MME-Realworld上提升5.4%,在挑战性的分布外(OOD)V* Bench上提升5.2%。值得注意的是,对Qwen2.5-VL-7B进行21K样本后训练后,其在OOD V* Bench上的表现超越OpenAI o1和GPT-4o模型。代码已开源。
原文摘要 · Abstract (English)
State-of-the-art large multi-modal models (LMMs) face challenges when processing high-resolution images, as these inputs are converted into enormous visual tokens, many of which are irrelevant to the downstream task. In this paper, we propose Multi-turn Grounding-based Policy Optimization (MGPO), an end-to-end reinforcement learning (RL) framework that enables LMMs to iteratively focus on key visual regions by automatically cropping sub-images, based on model-predicted grounding coordinates within a multi-turn conversation framework. Compared to supervised fine-tuning (SFT), which requires costly additional grounding annotations, our approach highlights that LMMs can emerge robust grounding abilities during the RL training process, leveraging only a binary reward function derived from the correctness of the final answer. Additionally, we observe that LMMs struggle to autonomously trigger visual grounding during the rollout process. To address this cold start problem, we design a multi-turn conversational template and restrict policy loss computation to model outputs generated across multiple dialogue rounds, thereby promoting stable optimization. Extensive experiments demonstrate that, when trained on standard visual-question-short answering data without grounding annotations, MGPO effectively elicits stronger grounding capabilities compared to GRPO, leading to 5.4\% improvement on in-distribution MME-Realworld and 5.2\% improvement on the challenging out-of-distribution (OOD) V* Bench. Notably, MGPO post-training on Qwen2.5-VL-7B with 21K samples surpasses OpenAI's o1 and GPT-4o models on the OOD V* Bench. Codes are available at https://github.com/EvolvingLMMs-Lab/MGPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。