arXiv:2609.03557cs.CVcs.GR2026-09

用虚幻引擎生成带动作控制的视频数据,解决世界模型训练数据难题

Building Pretraining Data for World Models: An Unreal Engine-Based Pipeline for Action-Conditioned Video Generation

论文配图:Building Pretraining Data for World Models: An Unreal Engine-Based Pipeline for Action-Conditioned Video Generation
图 1 · 摘自论文原文
  • 分两阶段生成:实时物理模拟记录轨迹,离线渲染输出高质量视频
  • 产出2691小时1080p和6076小时720p多视角视频,共429个场景可用
  • 支持分布式生产、自动筛选与错误恢复,适合大规模视觉世界模型训练

行动条件视频模型需要大量与控制信号时间对齐的视觉数据,但真实视频中动作原因通常未知。本文提出基于虚幻引擎的大规模合成数据生成流水线,通过两阶段设计实现:第一阶段在PIE中运行实时物理模拟,记录每帧角色状态、控制输入与相机状态;第二阶段在新进程中重放轨迹,使用Movie Render Queue(MRQ)离线渲染。系统包含25台服务器,每台配置8张NVIDIA RTX 5090 GPU,从2384个资产包中筛选出429个场景及40个拟人角色。已生成2691小时1080p视频与6076小时720p视频。系统支持缓存感知任务划分、节点本地调度、自动场景筛选、美学与亮度过滤、部分输出恢复、异步上传与集群健康监控。该流水线是EchoWM项目中使用的虚幻引擎合成数据核心组件。

原文摘要 · Abstract (English)

Action-conditioned video models require large-scale visual data paired with control signals that are temporally aligned with the resulting scene transitions. Such supervision is difficult to obtain from ordinary real-world video because the actions that caused each visual change are typically unknown. We present a large-scale synthetic data production pipeline built on Unreal Engine for generating action-conditioned, multi-view video. To accommodate the different execution requirements of real-time physics and high-quality offline rendering, the pipeline executes trajectory generation and final rendering in two stages: Stage I runs real physics in PIE and records per-frame character states, control inputs, and camera states into an intermediate trajectory representation; Stage II replays those trajectories in a new engine process and renders them offline with Movie Render Queue (MRQ). Around this core, we develop a distributed production system with cache-aware task partitioning, node-local slot scheduling, automated scene screening, aesthetic and luminance filtering, partial-output recovery, asynchronous upload, and continuous cluster health monitoring. The production cluster contains 25 servers with eight NVIDIA RTX 5090 GPUs per server. From 2,384 asset packs, 429 levels were retained for production together with a pool of 40 humanoid characters. The pipeline has produced 2,691 hours of 1080p video and 6,076 hours of 720p video. We describe the system architecture, the implementation decisions that emerged from production failures, and the limitations of using perceptual quality proxies for world-model data curation. The pipeline described in this report constitutes the Unreal Engine synthetic-data production component used in EchoWM.

世界模型合成数据虚幻引擎视频生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。