首个面向专业游戏引擎的项目级代码数据集与评测基准，填补了游戏开发AI研究空白。
JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Engines
- 基于游戏开发比赛开源项目构建，采用确定性验证流程筛选高质量代码。
- 大项目上模型运行通过率仅5.7%，暴露架构设计是主要瓶颈。
- 适合研究游戏生成、代码补全及智能开发工具的学者和工程师。
当前基于AI的游戏开发在资产生成、玩法设计和网页游戏编码方面取得显著进展，但专业游戏引擎上的项目级代码工程仍缺乏大规模数据集与可复现的评估方法。本文提出JamSet与JamBench，首个基于专业游戏引擎（Godot）的项目级代码框架数据集与评测基准。利用游戏创作大赛中数千个开源项目，结合文本格式与无头执行模式，设计从文件完整性到运行行为的确定性验证流水线，从超过24万仓库中提炼出8,133个已验证项目，其中300个经人工验证构成JamBench；其余为JamSet。JamBench定义主题驱动生成与代码补全任务，通过编译通过率、结构完整度评分（SCS）与行为对齐评分（BAS）综合评估。9个前沿模型测试显示，随着项目规模增大，运行通过率从小型项目80.4%骤降至大型项目5.7%（任务2a）。代码智能体虽提升编译成功率，但在运行行为质量上无改善，表明瓶颈在于架构设计而非语法正确性。实验验证了JamSet作为训练数据的有效性。所有数据与代码均公开可用。
原文摘要 · Abstract (English)
Current AI-driven game development has made substantial progress in asset generation, gameplay design, and web-based game coding, yet project-level code engineering on professional game engines remains largely unexplored due to the absence of large-scale datasets and deterministic evaluation methods. We present JamSet and JamBench, the first project-level game code framework dataset and benchmark built on a professional game engine. Our key insight is that Game Jam competitions, community events where developers build complete games under tight time constraints, yield thousands of open-source projects suitable for this purpose. Building on the Godot engine's text-based format and headless execution mode, we design a deterministic verification pipeline from file integrity to runtime behavior collection, distilling 8,133 verified projects from over 240,000 repositories. Of these, 300 manually verified projects form JamBench; the rest constitute JamSet. JamBench defines theme-driven generation and code completion tasks, evaluated through a pipeline combining compilation pass rates, Structural Completeness Score (SCS), and Behavioral Alignment Score (BAS). Evaluation of 9 frontier models reveals a capability cliff as project scale increases, with runtime pass rates dropping from 80.4% on small projects to 5.7% on large ones (Task2a). Code Agents improve compilation rates yet yield no gains in runtime behavioral quality, indicating that the bottleneck lies in architectural design rather than syntactic correctness. Experiments validate JamSet as effective training data. All data and code are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。