让视觉语言模型通过看视频反思游戏失败，自动改进策略。
GameVerse: Can Vision-Language Models Learn from Video-based Reflection?
- 用看视频反思+重试的循环评估模型学习能力。
- 结合失败轨迹和专家教程，性能接近强化学习加监督微调。
- 覆盖15款热门游戏，支持语义与界面双控制输入。
人类游戏行为是一种视觉驱动的互动循环：行动、反思失败、观看教程以优化策略。视觉语言模型（VLMs）能否也通过视频反思来学习？我们提出GameVerse，一个全面的游戏视频基准，支持反射式视觉交互循环。不同于传统的“一次性”评估，它采用新颖的“反思-重试”范式，评估VLM如何内化视觉经验并改进策略。为实现系统化与可扩展评估，我们引入涵盖15款全球热门游戏的认知层级分类体系，支持语义与图形用户界面（GUI）双重动作空间，并使用先进VLM进行里程碑评估以量化进展。实验表明，VLM在多种设置下均能从视频反思中获益，最佳效果来自结合失败轨迹与专家教程——这是一种无需训练的类强化学习加监督微调方法。项目主页见https://gameverse-bench.github.io/，代码开源于https://github.com/THUSI-Lab/GameVerse。
原文摘要 · Abstract (English)
Human gameplay is a visually grounded interaction loop in which players act, reflect on failures, and watch tutorials to refine strategies. Can Vision-Language Models (VLMs) also learn from video-based reflection? We present GameVerse, a comprehensive video game benchmark that enables a reflective visual interaction loop. Moving beyond traditional fire-and-forget evaluations, it uses a novel reflect-and-retry paradigm to assess how VLMs internalize visual experience and improve policies. To facilitate systematic and scalable evaluation, we also introduce a cognitive hierarchical taxonomy spanning 15 globally popular games, dual action space for both semantic and GUI control, and milestone evaluation using advanced VLMs to quantify progress. Our experiments show that VLMs benefit from video-based reflection in varied settings, and perform best by combining failure trajectories and expert tutorials-a training-free analogue to reinforcement learning (RL) plus supervised fine-tuning (SFT).Our project page is available at https://gameverse-bench.github.io/ . Our code is available at https://github.com/THUSI-Lab/GameVerse .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。