用强化学习让视觉语言模型学会在游戏里自主决策，表现超越顶级商用模型。
G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learning
- 通过自演化训练生成感知与推理能力，构建可扩展的多游戏训练环境
- 新模型在所有游戏中均超越教师模型，性能超过Claude-3.7-Sonnet-Thinking
- 发现感知与推理能力在训练中相互促进，适合想提升VLM交互能力的研究者
视觉语言模型（VLMs）在直接多模态任务中表现优异，但在需要决策的视觉丰富环境（如游戏）中表现不佳，存在‘知行鸿沟’。为此，本文提出VLM-Gym——一个包含多样视觉游戏、统一接口且难度可调的强化学习环境，支持大规模并行训练。基于该环境，我们训练了仅依赖强化学习的G0模型，其展现出涌现的感知与推理模式。为进一步应对游戏多样性挑战，提出G1模型，在强化学习微调前引入增强感知的冷启动先验。实验表明，G1模型在所有游戏上均优于教师模型，并超越主流闭源模型Claude-3.7-Sonnet-Thinking。系统分析揭示：感知与推理能力在强化学习过程中相互促进、协同进化。代码与环境已开源。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) excel in many direct multimodal tasks but struggle to translate this prowess into effective decision-making within interactive, visually rich environments like games. This ``knowing-doing'' gap significantly limits their potential as autonomous agents, as leading VLMs often performing badly in simple games. To address this, we introduce VLM-Gym, a curated reinforcement learning (RL) environment featuring diverse visual games with unified interfaces and adjustable, compositional difficulty, specifically designed for scalable multi-game parallel training. Leveraging VLM-Gym, we train G0 models using pure RL-driven self-evolution, which demonstrate emergent perception and reasoning patterns. To further mitigate challenges arising from game diversity, we develop G1 models. G1 incorporates a perception-enhanced cold start prior to RL fine-tuning. Our resulting G1 models consistently surpass their teacher across all games and outperform leading proprietary models like Claude-3.7-Sonnet-Thinking. Systematic analysis reveals an intriguing finding: perception and reasoning abilities mutually bootstrap each other throughout the RL training process. Source code including VLM-Gym and RL training are released at https://github.com/chenllliang/G1 to foster future research in advancing VLMs as capable interactive agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。