从文本生成带完整标注的多角色视频,世界状态可追溯、可验证。
The GEST-Engine: From Event Graphs to Synthetic Video. A Full Technical Report

- 用事件时空图(GEST)显式建模世界状态,确保可解释性。
- 单次模拟输出多模态数据,零边际标注成本,含骨骼、深度、关系图等。
- 适合做视频理解模型的训练数据与评估基准,支持大规模生成。
我们提出GEST-Engine,一个从自然语言生成全标注多角色视频的完整系统。核心是显式世界模型:不依赖学习的隐变量,而是通过形式化事件时空图(GEST)确定性地维护世界状态(包含角色、位置、动作、持物及事件时空间关系),在商用游戏引擎中运行,由开源多人脚本框架驱动。GEST可通过程序生成或由代理式文生图系统生成,该系统利用大模型规划故事并通过程序后端验证,保证生成规格可执行。随后进入四阶段流程:图解析与验证、实体与动作定位、时间编排(使用Floyd-Warshall传递闭包求解Allen约束)、执行与捕获。单次仿真输出帧对齐的RGB视频、密集像素级深度、实例分割、角色骨骼姿态、每帧的成对空间关系图、2D边界框、事件到帧的时间映射及自然语言描述,所有数据均无额外标注成本。系统还包含在游戏内世界编辑器、运行时能力提取、文本生成管道和跨并行虚拟机的大规模数据生成架构。因每帧均可回溯至语义规范,系统天然保证物体恒常性、多角色协同与时间一致性,其输出可用于视频理解模型的训练、评测基准与诊断工具。
原文摘要 · Abstract (English)
We present the GEST-Engine, a complete system that goes from natural-language text to fully-annotated multi-actor video. At its core is an explicit world model: rather than encoding state as a learned latent, the engine maintains a complete, inspectable representation of the world (which actors exist, where they are, what they are doing, which objects they hold, and how events relate in time and space), expressed as a formal Graph of Events in Space and Time (GEST) and realized deterministically inside the open world of a commercial game engine driven through an open-source multiplayer scripting framework. GESTs are produced either procedurally or by an agentic text-to-GEST system in which an LLM Director plans a story through tool calls validated by a programmatic state backend, so every generated specification is executable by construction. A GEST then enters a four-stage execution pipeline: graph parsing and validation, entity and action grounding, temporal orchestration (Allen-style constraints resolved by Floyd-Warshall transitive closure), and execution and capture. In a single simulation pass the engine emits frame-aligned RGB video, dense per-pixel depth, instance segmentation, per-actor skeletal pose, per-frame pairwise spatial-relation graphs, 2D bounding boxes, event-to-frame temporal mappings, and natural-language descriptions, all at zero marginal annotation cost. We further describe an in-game world editor, runtime capability extraction, a text-generation pipeline, and a production system that renders corpora at scale across parallel virtual machines. Because every frame traces back to a semantic specification, the engine guarantees object permanence, multi-actor coordination, and temporal consistency by construction, making its output valuable as training data, evaluation benchmarks, and diagnostic tools for video understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。