用小模型复现大模型的代码环境生成能力,让轻量LLM也能自动生成游戏规则代码。
Distilling Game Code World Model Generation into Lightweight Large Language Models

- 通过后训练将大模型的游戏代码生成能力迁移到小模型上
- 在30个游戏中验证,小模型生成代码的语法正确率与规则符合率显著提升
- 适合需要低成本自动构建智能体环境的研究者和开发者
大型语言模型(LLMs)在从自然语言生成可执行代码方面展现出强大能力,为自动构建智能体环境提供了可能。近期关于代码世界模型(CWMs)的研究表明,LLMs可将游戏规则转化为与蒙特卡洛树搜索等求解器兼容的Python实现。本文聚焦于游戏场景,要求生成的环境需完整实现规则、合法动作、状态转移、观测和奖励。这类特定游戏的可执行模型称为游戏代码世界模型(GameCWMs)。然而,当前方法依赖前沿模型与推理时的迭代优化,限制了可访问性与可扩展性。本文研究是否可通过后训练将GameCWM生成能力蒸馏至小型模型。提出:(1) 一个涵盖30个游戏的精选数据集,包含完美与不完美信息博弈;(2) 一套验证框架,用于评估生成代码在结构与语义层面是否符合游戏特性;(3) 结合监督微调(SFT)与可验证奖励强化学习(RLVR)的后训练流程。在Qwen2.5-3B-Instruct上实验发现,SFT可提升语法正确性,而RLVR能增强执行层面的规则遵循度,显著改善其在完美与不完美信息博弈中生成有效GameCWM的能力。整体而言,该流程使小模型具备更强的自动环境生成能力,为自然语言到环境的自动化提供了一条可扩展路径。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown great ability in generating executable code from natural language, opening the possibility of automatically constructing environments for AI agents. Recent work on Code World Models (CWMs) demonstrates that LLMs can translate game rules into Python implementations compatible with solvers like Monte Carlo Tree Search. We study this problem in game settings, where generated environments must implement rules, legal actions, state transitions, observations, and rewards. We refer to these game-specific executable models as Game Code World Models (GameCWMs). However, current approaches to generating code world models rely on frontier models and inference-time refinement loops, limiting accessibility and scalability. This work investigates whether GameCWM generation capabilities can be distilled into smaller models through post-training. We introduce: (1) a curated dataset of 30 games spanning perfect and imperfect information games, (2) a verification framework that evaluates generated code against structural and semantic game properties, and (3) a post-training pipeline combining Supervised Fine-Tuning (SFT) with Reinforcement Learning with Verifiable Rewards (RLVR). We experiment with Qwen2.5-3B-Instruct and find that SFT can increase syntactic correctness, while RLVR can improve execution-level adherence to game rules, thereby improving Qwen's ability to generate valid GameCWMs in both perfect and imperfect information games. Overall, our pipeline makes Qwen2.5-3B-Instruct more capable of generating valid GameCWMs, thereby offering a scalable path toward automatic environment generation from natural language.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。