大模型生成的游戏规则虽预测准确，却可能因遗漏关键规则导致游戏失败。
When a Verified World Model Still Loses: Play-Adequacy vs Prediction-Accuracy in LLM-Synthesized Code World Models
- 用代码构建世界模型，但仅靠采样准确性无法保证规划有效
- 遗漏<1%的规则就导致游戏损失，代价为0.091（置信区间[0.065,0.117]）
- 适合关注规划型世界模型评估标准的研究者
大型语言模型可将游戏规则合成可执行代码——代码世界模型（CWM），供经典规划器搜索。现有方法通常以采样轨迹上的高转移准确率为验收标准。本文指出此标准对规划任务而言不充分。我们证明四点：(1) 即使在采样轨迹上达到100%转移准确率、规划搜索分布下状态准确率≥98%，模型仍会系统性失败，因其遗漏的关键动态占不到1%，但导致的博弈损失为0.091（种子聚类95%置信区间[0.065,0.117]，n=4800）。该现象称为“验证-正确性差距”，并在全合成流程中端到端验证。(2) 损害遵循定量规律：danger = play_cost × (1−rarity)^N，其中(1−rarity)^N为门控漏失因子，已严格证明，且play_cost可实证界定。(3) 增加数据无法修复问题：LLM合成表现为规则翻译而非推理，不同模型（GPT-5.x）与数据范式（包括DAgger和靶向示例）均未推断出缺失规则。(4) 该机制同样存在于不完全信息CWM的信念推理函数中：我们证明覆盖界限（大小为N的门控需满足N≳b^{d_max}），解释为何浅层游戏如Kuhn扑克无此差距；并手构造Beacon——一个通过验证但推理错误的函数，其表现每局必败。结果表明，规划导向世界模型的适切性应基于搜索分布或直接通过博弈表现衡量，而非采样轨迹上的预测准确率。
原文摘要 · Abstract (English)
Large language models can synthesize a game's rules as executable code - a Code World Model (CWM) - which a classical planner then searches over. Such models are typically accepted when they reach high transition accuracy on sampled trajectories. We argue this is the wrong notion of adequacy for planning. We show four things. (1) An LLM-synthesized CWM can pass a sampling gate at 100% transition accuracy and be $\geq 98\%$ state-accurate on the planner's own search distribution, yet lose systematically at play, because the $<1\%$ it gets wrong is exactly the pivotal dynamics; the play cost of the omitted rule is $0.091$ (seed-clustered 95% CI $[0.065,0.117]$, $n=4800$). We call this the verified-vs-correct gap, and confirm it end-to-end through the synthesis pipeline. (2) The harm follows a quantitative law, $\mathrm{danger}=\mathrm{play\_cost}\times(1-\mathrm{rarity})^N$, whose $(1-\mathrm{rarity})^N$ gate-miss factor is proven exact and whose play cost is empirically bounded. (3) The failure is not repaired by more data: LLM synthesis behaves as rule translation, not rule inference, and did not infer the omitted rule across models (GPT-5.x) and data regimes (including DAgger and targeted examples). (4) The same mechanism recurs on the belief-inference function of imperfect-information CWMs: we prove a coverage bound (a size-$N$ gate is identifying when $N\gtrsim b^{d_{\max}}$), explaining why shallow games such as Kuhn poker show no gap, and hand-construct Beacon, a verified-but-wrong inference function that passes the gate yet loses every game. These results suggest adequacy for planning-oriented world models should be measured on the search distribution or by play directly, not by prediction accuracy on sampled transitions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。