仅用图像训练大模型,实现复杂视觉推理与自修正。
Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedback
- 构建推理-渲染-视觉反馈闭环,通过图像自监督优化
- 在图表与网页生成任务中超越同类开源模型
- 无需图文配对数据,适合无标注图像场景
多模态大语言模型(MLLM)在各类视觉任务中表现优异。然而,其深入视觉推理能力的发展受限于对精心标注的图文数据的依赖。为此,我们提出新颖框架「推理-渲染-视觉反馈」(RRVF),使MLLM仅通过原始图像学习复杂视觉推理。该框架基于「验证不对称性」原则:将生成结果与源图像对比验证,远比从零开始进行深度视觉推理生成结构化表示(如代码)简单。这一差异为强化学习提供了理想奖励信号,显著降低对图文监督的依赖。RRVF采用包含推理、渲染和视觉反馈的闭环迭代流程,支持多轮交互式自修正,并通过GRPO算法端到端优化。在数据图表与网页界面两个领域进行的广泛评估表明,经RRVF训练的模型不仅优于现有同规模开源MLLM及监督微调基线,还展现出更强泛化能力。值得注意的是,其性能超过训练中用于生成视觉反馈的更先进MLLM。代码已公开于https://github.com/L-O-I/RRVF。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) exhibit impressive performance across various visual tasks. Subsequent investigations into enhancing their visual reasoning abilities have significantly expanded their performance envelope. However, a critical bottleneck in the advancement of MLLMs toward deep visual reasoning is their heavy reliance on curated image-text supervision. To solve this problem, we introduce a novel framework, ``Reasoning-Rendering-Visual-Feedback'' (RRVF), that enables MLLMs to learn complex visual reasoning from only raw images. This framework builds on the ``Asymmetry of Verification'' principle, i.e., verifying the rendered output against the source image is substantially easier than performing deep visual reasoning to generate a faithful, structured representation such as code. We demonstrate that this relative ease provides an ideal reward signal for optimization via Reinforcement Learning (RL), thereby reducing reliance on image-text supervision. RRVF implements a closed-loop iterative process encompassing reasoning, rendering, and visual feedback components, enabling the model to perform complex reasoning, including self-correction through multi-turn interactions. This process is optimized end-to-end using the GRPO algorithm. Extensive evaluations are conducted on image-to-code generation across two diverse domains: data charts and web interfaces. The RRVF-trained model not only outperforms existing similarly sized open-source MLLMs and supervised fine-tuning baselines but also exhibits superior generalization. Notably, the model outperforms the more advanced MLLM used to generate visual feedback during training. Code is available at https://github.com/L-O-I/RRVF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。