用游戏评测大模型能力，解决感知差、提示敏感等问题。
lmgame-Bench: How Good are LLMs at Playing Games?
- 构建统一API的游戏评测平台，配轻量感知记忆框架。
- 13个主流模型在平台上表现差异明显，挑战性强。
- 单游戏强化学习可迁移至新游戏和规划任务，具泛化潜力。
视频游戏需要感知、记忆与规划能力，正是现代大语言模型代理应掌握的核心素养。我们研究了用主流视频游戏评估大模型时的主要挑战，发现直接将大模型投入游戏中无法有效评估，原因有三：视觉感知脆弱、提示敏感性高、存在潜在数据污染。为此提出lmgame-Bench，将平台跳跃、解谜与叙事类游戏整合为统一Gym风格接口，并配备轻量级感知与记忆支架，以稳定提示波动并消除污染。在13个领先模型上测试显示，lmgame-Bench既具挑战性又能良好区分模型性能。相关性分析表明，每款游戏均探测独特的能力组合，常被其他评测孤立考察。更有趣的是，在lmgame-Bench单一游戏中进行强化学习后，模型可迁移到未见游戏及外部规划任务。评估代码已开源。
原文摘要 · Abstract (English)
Playing video games requires perception, memory, and planning, exactly the faculties modern large language model (LLM) agents are expected to master. We study the major challenges in using popular video games to evaluate modern LLMs and find that directly dropping LLMs into games cannot make an effective evaluation, for three reasons -- brittle vision perception, prompt sensitivity, and potential data contamination. We introduce lmgame-Bench to turn games into reliable evaluations. lmgame-Bench features a suite of platformer, puzzle, and narrative games delivered through a unified Gym-style API and paired with lightweight perception and memory scaffolds, and is designed to stabilize prompt variance and remove contamination. Across 13 leading models, we show lmgame-Bench is challenging while still separating models well. Correlation analysis shows that every game probes a unique blend of capabilities often tested in isolation elsewhere. More interestingly, performing reinforcement learning on a single game from lmgame-Bench transfers both to unseen games and to external planning tasks. Our evaluation code is available at https://github.com/lmgame-org/GamingAgent/lmgame-bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。