让视觉语言模型像人一样动态聚焦图像重点,提升推理效率。
Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMs
- 通过动态视觉搜索与缩放,让模型自适应关注图像关键区域。
- 在多分辨率下提升5%准确率,尤其在强视觉推理任务上表现突出。
- 适合需要高效推理的视觉语言模型部署场景。
视觉语言模型(VLM)在多种计算机视觉任务中已取得显著性能。然而,现有模型对多模态推理能力尚未充分挖掘。本文提出链式焦点(Chain-of-Focus, CoF)方法,使VLM能够基于视觉线索和问题内容,自适应地聚焦并放大图像关键区域,实现高效的多模态推理。为实现该能力,我们设计了两阶段训练流程:监督微调(SFT)与强化学习(RL)。在SFT阶段,构建包含3000个样本的MM-CoF数据集,源自一个视觉代理,其能根据不同图像分辨率和问题自适应识别关键区域以完成视觉任务。使用该数据集对Qwen2.5-VL模型进行冷启动微调。在RL阶段,利用准确率与输出格式作为奖励信号,更新模型,进一步优化搜索与推理策略,无需人类先验。模型在多个基准测试中取得显著提升,在要求强视觉推理能力的V*基准上,相比现有VLM在8种图像分辨率(224至4K)下平均提升5%,验证了所提CoF方法的有效性,推动了VLM在实际应用中的高效部署。
原文摘要 · Abstract (English)
Vision language models (VLMs) have achieved impressive performance across a variety of computer vision tasks. However, the multimodal reasoning capability has not been fully explored in existing models. In this paper, we propose a Chain-of-Focus (CoF) method that allows VLMs to perform adaptive focusing and zooming in on key image regions based on obtained visual cues and the given questions, achieving efficient multimodal reasoning. To enable this CoF capability, we present a two-stage training pipeline, including supervised fine-tuning (SFT) and reinforcement learning (RL). In the SFT stage, we construct the MM-CoF dataset, comprising 3K samples derived from a visual agent designed to adaptively identify key regions to solve visual tasks with different image resolutions and questions. We use MM-CoF to fine-tune the Qwen2.5-VL model for cold start. In the RL stage, we leverage the outcome accuracies and formats as rewards to update the Qwen2.5-VL model, enabling further refining the search and reasoning strategy of models without human priors. Our model achieves significant improvements on multiple benchmarks. On the V* benchmark that requires strong visual reasoning capability, our model outperforms existing VLMs by 5% among 8 image resolutions ranging from 224 to 4K, demonstrating the effectiveness of the proposed CoF method and facilitating the more efficient deployment of VLMs in practical applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。