用自动生成的游戏评估大模型的通用推理能力
Measuring General Intelligence with Generated Games
- 用大模型生成新游戏描述并转为代码环境，动态构造评测场景
- GPT-4o等顶尖模型在新游戏上胜率仅7-9%，推理模型达31-36%
- 适合研究通用智能评估、强化学习与语言模型融合的学者
我们提出gg-bench，一个用于评估语言模型通用推理能力的游戏环境集合。与传统静态基准不同，gg-bench是一个数据生成过程，可随时生成新评测实例。具体通过：(1) 使用大语言模型（LLM）生成新颖游戏的自然语言描述；(2) 用同一LLM将游戏实现为Gym环境；(3) 在生成游戏中训练强化学习（RL）代理进行自对弈。评估时，将游戏描述、当前棋盘状态和合法动作列表作为提示输入语言模型，模型输出其选择的动作。gg-bench极具挑战性：如GPT-4o和Claude 3.7 Sonnet等前沿模型在上下文学习下胜率仅为7-9%；而推理模型如o1、o3-mini和DeepSeek-R1平均胜率达31-36%。我们公开生成的游戏、数据生成流程及评估代码，以支持后续建模研究与基准拓展。
原文摘要 · Abstract (English)
We present gg-bench, a collection of game environments designed to evaluate general reasoning capabilities in language models. Unlike most static benchmarks, gg-bench is a data generating process where new evaluation instances can be generated at will. In particular, gg-bench is synthetically generated by (1) using a large language model (LLM) to generate natural language descriptions of novel games, (2) using the LLM to implement each game in code as a Gym environment, and (3) training reinforcement learning (RL) agents via self-play on the generated games. We evaluate language models by their winrate against these RL agents by prompting models with the game description, current board state, and a list of valid moves, after which models output the moves they wish to take. gg-bench is challenging: state-of-the-art LLMs such as GPT-4o and Claude 3.7 Sonnet achieve winrates of 7-9% on gg-bench using in-context learning, while reasoning models such as o1, o3-mini and DeepSeek-R1 achieve average winrates of 31-36%. We release the generated games, data generation process, and evaluation code in order to support future modeling work and expansion of our benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。