用游戏开发生成可验证的轨迹数据,提升世界模型训练效率
Agentic Game Development as a Verifiable Trajectory Data Engine for Scaling World Models

- 将游戏开发作为可执行的环境,利用引擎自动反馈碰撞、物理等信号
- 结合人类验收反馈,构建密集且可信的强化学习奖励信号
- 适合需要长时序、高精度空间建模的智能体训练场景
当前世界模型的扩展依赖于更多爬取视频和更大算力,但该策略效率低下。世界模型的真正扩展需要递归的数据引擎,提供基于事实的奖励信号。代码代理的成功说明了这一点:代码可执行,编译器和运行时可提供高质量的强化学习(RL)后训练奖励。相比之下,空间生成仍依赖模糊的代理指标(如CLIP分数),这些信号不精确且有偏差,难以支持强化学习后训练。而游戏开发提供了缺失的奖励环境:游戏引擎编码的场景是可执行的世界规范,能高效检测碰撞、物理合理性、可导航性及可玩性边界;开发者通过判断场景是否接受,提供全局验证信号。此外,游戏开发还产生真实世界的长时序轨迹数据,用于强化学习后训练。因此,我们提出一种强化学习与人-引擎验证结合(RLHEV)的后训练范式,融合密集引擎信号与开发过程中的隐式人类接受反馈。
原文摘要 · Abstract (English)
A common strategy for scaling world models is to train on more crawled video with more compute. We argue that this strategy is inefficient: scaling world models also requires a recursive data engine that offers grounded reward signals. The success of code agents illustrates why this matters. As code is executable, compilers and runtimes can provide high-quality rewards for Reinforcement Learning (RL) post-training of LLMs. By contrast, spatial generation still relies largely on fuzzy proxies such as CLIP scores. These signals are fuzzy and biased, making them hard to support RL post-training. Compared with these, game development provides a missing reward environment for spatial world models. A scene encoded by a game engine is an executable world specification: the engine can efficiently check collision, physics, navigability and bounded playability, while the developer provides the global verification signal by judging whether the scene should be accepted. Game development also provides real-world long-horizon trajectory data for RL post-training. We therefore propose Reinforcement Learning with Human-Engine Verification (RLHEV), a post-training paradigm that combines dense engine signals with implicit human acceptance feedback from the development process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。