用视觉模型玩《黑神话:悟空》,90%的中低难度战斗能通关。
Can VLMs Play Action Role-Playing Games? Take Black Myth Wukong as a Study Case
- 构建视觉动作角色扮演框架,仅凭画面理解执行复杂操作。
- 在12个任务中,90%的中低难度战斗成功通关。
- 首次公开人类操作数据集,推动多模态游戏智能研究。
近期基于大语言模型(LLM)的智能体在多个领域取得进展,尤其在视频游戏应用方面备受关注。传统方法依赖游戏API获取环境与动作数据,受限于API可用性,且无法模拟真实玩家行为。随着视觉语言模型(VLMs)的发展,智能体具备更强的视觉理解能力,可仅通过图像输入与游戏交互。然而,现有方法在动作导向任务中仍面临挑战,尤其在动作角色扮演游戏(ARPG)中,强化学习方法泛化能力差且需大量训练。为此,本文以ARPG《黑神话:悟空》为研究平台,探索当前VLM在纯视觉输入与复杂动作输出场景下的能力边界。定义12项任务,其中75%聚焦战斗,引入多个先进VLM进行评估。同时发布包含游戏操作日志与鼠标键盘记录的人类操作数据集。提出新型VARP(Vision Action Role-Playing)智能体框架,包含动作规划系统与视觉轨迹系统,在易、中等难度战斗场景中成功率达90%。本研究为多模态智能体在复杂动作游戏中的应用提供新思路。代码与数据集将开源。
原文摘要 · Abstract (English)
Recently, large language model (LLM)-based agents have made significant advances across various fields. One of the most popular research areas involves applying these agents to video games. Traditionally, these methods have relied on game APIs to access in-game environmental and action data. However, this approach is limited by the availability of APIs and does not reflect how humans play games. With the advent of vision language models (VLMs), agents now have enhanced visual understanding capabilities, enabling them to interact with games using only visual inputs. Despite these advances, current approaches still face challenges in action-oriented tasks, particularly in action role-playing games (ARPGs), where reinforcement learning methods are prevalent but suffer from poor generalization and require extensive training. To address these limitations, we select an ARPG, ``Black Myth: Wukong'', as a research platform to explore the capability boundaries of existing VLMs in scenarios requiring visual-only input and complex action output. We define 12 tasks within the game, with 75% focusing on combat, and incorporate several state-of-the-art VLMs into this benchmark. Additionally, we will release a human operation dataset containing recorded gameplay videos and operation logs, including mouse and keyboard actions. Moreover, we propose a novel VARP (Vision Action Role-Playing) agent framework, consisting of an action planning system and a visual trajectory system. Our framework demonstrates the ability to perform basic tasks and succeed in 90% of easy and medium-level combat scenarios. This research aims to provide new insights and directions for applying multimodal agents in complex action game environments. The code and datasets will be made available at https://varp-agent.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。