让视觉语言模型通过自生成游戏自动进化,无需人工标注。
Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play
- 用多人角色对抗游戏训练模型,自动产生无标签训练数据。
- 从任意图片生成游戏,跨场景推理能力显著提升。
- 自迭代优化算法突破性能瓶颈,适合大规模模型持续进化。
尽管强化学习为视觉语言模型和多模态大模型的改进提供了新路径,但现有方法严重依赖人工标注数据与昂贵的人工验证,限制了多模态系统的可扩展自进化。为此,我们提出 Vision-Zero——一种无标签、领域无关的多智能体自对弈框架,通过任意图像生成的视觉游戏实现视觉语言模型的自主演进。该框架具备三大特性:(1)策略性自对弈机制:在类似“谁是卧底”的游戏中训练模型进行多角色策略推理与交互,模型自主生成训练数据;(2)任意图像生成游戏:不依赖特定图像类型,可处理基于 CLEVR 的合成场景、图表及真实图像,增强跨任务泛化能力;(3)可持续性能提升:引入迭代自对弈策略优化(Iterative-SPO),交替执行自对弈与可验证奖励强化学习(RLVR),有效打破自对弈训练中的性能停滞,实现长期稳定提升。即使使用无标签数据,Vision-Zero 在推理、图表问答和视觉理解任务上均达到当前最优水平,优于依赖人工标注的方法。代码与模型已开源于 https://github.com/wangqinsi1/Vision-Zero。
原文摘要 · Abstract (English)
Although reinforcement learning (RL) has emerged as a promising approach for improving vision-language models (VLMs) and multimodal large language models (MLLMs), current methods rely heavily on manually curated datasets and costly human verification, which limits scalable self-improvement in multimodal systems. To address this challenge, we propose Vision-Zero, a label-free, domain-agnostic multi-agent self-play framework for self-evolving VLMs through competitive visual games generated from arbitrary image inputs. Specifically, Vision-Zero encompasses three main attributes: (1) Strategic Self-Play Framework: Vision-Zero trains VLMs in "Who Is the Spy"-style games, where the models engage in strategic reasoning and actions across multiple roles. Through interactive gameplay, models autonomously generate their training data without human annotation. (2) Gameplay from Arbitrary Images: Unlike existing gamified frameworks, Vision-Zero can generate games from arbitrary images, thereby enhancing the model's reasoning ability across diverse domains and showing strong generalization to different tasks. We demonstrate this versatility using three distinct types of image datasets: CLEVR-based synthetic scenes, charts, and real-world images. (3) Sustainable Performance Gain: We introduce Iterative Self-Play Policy Optimization (Iterative-SPO), a novel training algorithm that alternates between Self-Play and reinforcement learning with verifiable rewards (RLVR), mitigating the performance plateau often seen in self-play-only training and achieving sustained long-term improvements. Despite using label-free data, Vision-Zero achieves state-of-the-art performance on reasoning, chart question answering, and vision-centric understanding tasks, surpassing other annotation-based methods. Models and code have been released at https://github.com/wangqinsi1/Vision-Zero.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。