arXiv:2510.16333cs.CVcs.LG2025-10被引 2

强化学习让多模态模型看图更准,视觉表征更精准。

RL makes MLLMs see better than SFT

  • 用强化学习替代监督微调,提升多模态模型视觉理解能力。
  • 强化学习使视觉编码器产生更强且定位更精确的图像表征。
  • 新方法PIVOT仅需不到1%算力,效果超越大型预训练模型。

多模态语言模型(MLLM)研究中普遍认为其性能主要依赖大语言模型主干,导致对视觉编码器的理解严重不足。近期从监督微调(SFT)转向强化学习(RL)的训练范式,进一步加剧了这一问题——缺乏对训练方式如何重塑视觉编码器的深入分析。本文系统比较不同训练策略对MLLM的影响,发现RL在强视觉相关的VQA基准上显著优于SFT。通过从ImageNet分类、分割到梯度可视化等多维度实验,揭示训练策略不仅影响下游任务表现,更从根本上重构了模型的视觉表征。核心发现:相比SFT,RL生成更强大且空间定位更精确的视觉表示。据此提出简单高效的优化方案PIVOT(Preference-Instructed Vision OpTimization),集成后视觉编码器在极低计算成本下(<1%标准预训练算力)表现超越更大更重的对手。该结果为高效提升MLLM视觉骨干提供了可行路径。

原文摘要 · Abstract (English)

A dominant assumption in Multimodal Language Model (MLLM) research is that its performance is largely inherited from the LLM backbone, given its immense parameter scale and remarkable capabilities. This has created a void in the understanding of the vision encoder, which determines how MLLMs perceive images. The recent shift in MLLM training paradigms, from Supervised Finetuning (SFT) to Reinforcement Learning (RL), magnifies this oversight-namely, the significant lack of analysis on how such training reshapes the vision encoder as well as the MLLM. To address this, we first investigate the impact of training strategies on MLLMs, where RL shows a clear advantage over SFT in strongly vision-related VQA benchmarks. Motivated by this, we conduct a critical yet under-explored analysis of the vision encoder of MLLMs through diverse and in-depth experiments, ranging from ImageNet classification and segmentation to gradient visualization. Our results demonstrate that MLLM's post-training strategy (i.e., SFT or RL) not only leads to distinct outcomes on MLLM downstream tasks, but also fundamentally reshapes MLLM's underlying visual representations. Specifically, the key finding of our study is that RL produces stronger and precisely localized visual representations compared to SFT, boosting the ability of the vision encoder for MLLM. We then reframe our findings into a simple recipe for building strong vision encoders for MLLMs, Preference-Instructed Vision OpTimization (PIVOT). When integrated into MLLMs, a PIVOT-trained vision encoder outperforms even larger and more heavily-trained counterparts, despite requiring less than 1% of the computational cost of standard vision pretraining. This result opens an effective and efficient path for advancing the vision backbones of MLLMs. Project page available at https://june-page.github.io/pivot/

多模态强化学习视觉表征高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。