用四维评估框架揭示大模型生成游戏场景的虚假通过率
Mage: Multi-Axis Evaluation of LLM-Generated Executable Game Scenes Beyond Compile-Pass Rate
- 设计四轴评估体系：编译通过、运行成功、结构保真、机制遵循
- 直接生成仅43%运行通过，但结构空洞；加结构约束后结构完整度达1.00
- 输入粒度细化到行为级已无提升，说明当前条件已达饱和
编译通过率是大模型代码生成的主要评估指标，但在多组件领域特定产物中可能具有误导性。本文在可执行游戏场景合成任务中提出四轴评估协议（名为Mage）：编译成功、运行成功、结构保真度、机制遵循度。对四个开源大模型（7B–30B）、26个手工设计的Unity目标模式、两种自动提取的IR粒度级别，共进行了858次生成实验。直接从自然语言生成C#代码的平均运行通过率为43%，但结构空洞（机制F₁≈0.12）。引入结构化IR条件后，运行通过率减半，但结构保真度提升至1.00。在结构条件内，行为级与全场景粒度无统计差异（McNemar检验p=1.0），表明输入粒度已达到饱和。结果表明，在该领域编译通过率与功能正确性呈负相关，必须采用多维度评估。论文发布基准数据、回放日志及每条记录的指标供独立验证。
原文摘要 · Abstract (English)
Compile-pass rate is the dominant evaluation signal for LLM code generation, yet for multi-component domain-specific artifacts it can be actively misleading. We demonstrate this on executable game scene synthesis with a four-axis evaluation protocol (named `Mage') -- compile success, runtime success, structural fidelity, and mechanism adherence -- applied to 858 generation attempts across four open-weight LLMs (7B--30B), 26~hand-crafted Unity goal pattern playable concepts, and two automatically extracted IR granularity levels. Direct NL-to-C\# generation achieves the highest runtime-pass rate (43\% mean) yet produces structurally vacuous scenes (mechanism $F_1 \approx 0.12$). Structural IR conditioning halves the runtime rate but recovers domain-faithful structure ($F_1$ up to 1.00). Within IR conditioning, behavior-only and full-scene granularity are statistically indistinguishable (McNemar $p = 1.0$), indicating input-level granularity saturation. These results show that compile rate is anti-correlated with functional correctness in this domain and that multi-axis evaluation is necessary to detect the divergence. We release the benchmark, replay logs, and per-record metrics for independent verification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。