让视觉语言模型通过自生成问题自动提升推理能力。
VisPlay: Self-Evolving Vision-Language Models from Images
- 模型分角色互训：提问者出题，推理者答问，协同优化
- 在8个基准上提升推理能力，减少幻觉，泛化性能更好
- 无需人工标注，适合大规模多模态模型自我进化
强化学习为提升视觉语言模型在复杂推理任务中的表现提供了合理框架。然而，现有方法常依赖人工标注标签或特定任务启发式规则定义可验证奖励，成本高且难以扩展。我们提出VisPlay，一个自演化强化学习框架，使视觉语言模型能利用大量未标注图像数据自主改进推理能力。从单一基础模型出发，VisPlay将模型分为两个交互角色：图像条件提问者生成具有挑战性但可回答的视觉问题，多模态推理者生成银质答案。两者通过群体相对策略优化（GRPO）联合训练，引入多样性与难度奖励，平衡问题复杂度与答案质量。VisPlay在两种模型族上高效扩展。在Qwen2.5-VL和MiMo-VL上训练后，于包括MM-Vet和MMMU在内的八个基准上一致提升视觉推理、组合泛化能力并减少幻觉，展示了通往自演化多模态智能的可扩展路径。
原文摘要 · Abstract (English)
Reinforcement learning (RL) provides a principled framework for improving Vision-Language Models (VLMs) on complex reasoning tasks. However, existing RL approaches often rely on human-annotated labels or task-specific heuristics to define verifiable rewards, both of which are costly and difficult to scale. We introduce VisPlay, a self-evolving RL framework that enables VLMs to autonomously improve their reasoning abilities using large amounts of unlabeled image data. Starting from a single base VLM, VisPlay assigns the model into two interacting roles: an Image-Conditioned Questioner that formulates challenging yet answerable visual questions, and a Multimodal Reasoner that generates silver responses. These roles are jointly trained with Group Relative Policy Optimization (GRPO), which incorporates diversity and difficulty rewards to balance the complexity of generated questions with the quality of the silver answers. VisPlay scales efficiently across two model families. When trained on Qwen2.5-VL and MiMo-VL, VisPlay achieves consistent improvements in visual reasoning, compositional generalization, and hallucination reduction across eight benchmarks, including MM-Vet and MMMU, demonstrating a scalable path toward self-evolving multimodal intelligence. The project page is available at https://bruno686.github.io/VisPlay/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。