arXiv:2505.24025cs.CVcs.AI2025-05被引 3

用强化学习让视觉模型学会推理,首次实现视觉上下文推理能力提升

DINO-R1: Incentivizing Reasoning Capability in Vision Foundation Models

  • 提出GRQO方法,基于分组归一化对齐质量计算查询奖励
  • 在COCO、LVIS等数据集上显著优于监督微调基线
  • 适合研究视觉推理与提示工程的学者参考

大语言模型如DeepSeek-R1通过强化学习微调展现出强大推理能力,但视觉基础模型(如DINO系列)的推理能力仍被忽视。本文提出首个基于强化学习激励视觉上下文推理能力的方法——DINO-R1。DINO-R1引入一种新型训练策略Group Relative Query Optimization(GRQO),针对基于查询的表示模型,依据分组归一化的对齐质量计算查询级奖励,并采用KL正则化稳定物体性分布以缓解训练不稳定性。该联合优化机制实现了跨查询的密集且丰富的监督,同时抑制过拟合与分布漂移。基于Grounding-DINO构建的DINO-R1系列模型融合了视觉提示编码器与视觉引导的查询选择机制。在COCO、LVIS和ODinW上的大量实验表明,DINO-R1显著优于监督微调基线,在开放词汇与封闭集视觉提示场景中均表现出优异泛化能力。

原文摘要 · Abstract (English)

The recent explosive interest in the reasoning capabilities of large language models, such as DeepSeek-R1, has demonstrated remarkable success through reinforcement learning-based fine-tuning frameworks, exemplified by methods like Group Relative Policy Optimization (GRPO). However, such reasoning abilities remain underexplored and notably absent in vision foundation models, including representation models like the DINO series. In this work, we propose \textbf{DINO-R1}, the first such attempt to incentivize visual in-context reasoning capabilities of vision foundation models using reinforcement learning. Specifically, DINO-R1 introduces \textbf{Group Relative Query Optimization (GRQO)}, a novel reinforcement-style training strategy explicitly designed for query-based representation models, which computes query-level rewards based on group-normalized alignment quality. We also apply KL-regularization to stabilize the objectness distribution to reduce the training instability. This joint optimization enables dense and expressive supervision across queries while mitigating overfitting and distributional drift. Building upon Grounding-DINO, we train a series of DINO-R1 family models that integrate a visual prompt encoder and a visual-guided query selection mechanism. Extensive experiments on COCO, LVIS, and ODinW demonstrate that DINO-R1 significantly outperforms supervised fine-tuning baselines, achieving strong generalization in both open-vocabulary and closed-set visual prompting scenarios.

视觉推理强化学习提示工程模型微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。