用大模型生成游戏代码，测试发现小模型组合更靠谱。
From Code to Play: Benchmarking Program Search for Games Using Large Language Models
- 用大模型控制程序变异，进化出可用游戏代码
- 29个任务中，小模型组合效果优于单一大模型
- 适合想低成本试错的开发者或教育场景
大型语言模型（LLMs）在生成程序代码方面表现出色，为游戏中的程序合成提供了新机遇。本文探索了LLMs直接生成适用于多种游戏应用的代码潜力，聚焦于Python和Java两种语言。采用进化式爬山算法，由大模型控制初始程序的种子和突变。Python框架涵盖五款迷你Atari游戏、十关Baba is You、一个受Asteroids启发的环境以及迷宫生成任务；Java框架包含12个来自TAG桌游框架的游戏。在29个任务上，评估了12个Python模型和8个Java模型。结果表明，模型表现更依赖任务而非规模：更大模型虽生成更多可执行程序，但质量不升反降，且成本更高。无模型全面领先，但在特定任务上某模型可能更优。对同一问题尝试多个模型并取最佳结果，比单用一个模型更可靠。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown impressive capabilities in generating program code, opening exciting opportunities for applying program synthesis to games. In this work, we explore the potential of LLMs to directly synthesize usable code for a wide range of gaming applications, focusing on two programming languages, Python and Java. We use an evolutionary hill-climbing algorithm, where the mutations and seeds of the initial programs are controlled by LLMs. For Python, the framework covers various game-related tasks, including five miniature versions of Atari games, ten levels of Baba is You, an environment inspired by Asteroids, and a maze generation task. For Java, the framework contains 12 games from the TAG tabletop games framework. Across 29 tasks, we evaluated 12 language models for Python and 8 for Java. Our findings suggest that the performance of LLMs depends more on the task than on model size. While larger models generate more executable programs, these do not always result in higher-quality solutions but are much more expensive. No model has a clear advantage, although on any specific task, one model may be better. Trying many models on a problem and using the best results across them is more reliable than using just one.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。