将游戏视频中的界面元素自动去除，提升世界模型训练效果。
Game2World Engine: Unlocking In-the-Wild Gameplay Videos for World Model Training
- 构建全栈框架G2WEngine，自动提取并合成游戏界面
- 生成96K组对视频和1079段真实游戏片段用于训练与评估
- 提出无需掩码的GameCleaner模型，界面移除准确率超95%
视频游戏为视频世界模型提供了可扩展的训练数据，涵盖多样环境、复杂交互和丰富的在野游戏视频。然而，原始游戏画面将游戏世界与屏幕界面混杂，引入特定游戏偏差和无关动态，阻碍世界模型训练。为此，我们提出GameUI-Taxonomy与G2WEngine，一个完整的框架，形式化游戏界面定位与移除。G2WEngine从真实游戏视频中自动提取可复用的界面资产，并在干净画面中合成时间一致的界面叠加。基于此，我们构建Game2World数据集，包含96,000个合成配对视频（含精确重建目标）和1,079段来自303款游戏的真实在野片段，用于真实评估。其资产库包含5,132个经验证的界面元素，覆盖21个分类类别，源自1,010个代表性游戏帧。基于Game2World，我们提出GameCleaner，一种无掩码的游戏界面移除模型，结合多模态语义理解与视频编辑能力。相比依赖掩码的方法，GameCleaner直接识别并移除各类HUD元素，同时保留底层场景内容与时间动态。在受控实验中，使用去界面数据训练的世界模型，总体VideoReward提升6.83%。在去界面评估中，GameCleaner在合成视频上平均AAR达95.36%，优于最强时序掩码基线57.3%；在真实视频上取得80.05的AAR，背景保留率达99.8%。结果表明，该方法可规模化将互联网游戏视频转化为高质量世界模型训练数据。代码、数据集与模型将开源于https://github.com/Dongping-Chen/Game2World。
原文摘要 · Abstract (English)
Video games provide a scalable source of training data for video world models, offering diverse environments, complex interactions, and abundant in-the-wild gameplay videos. However, raw gameplay footage entangles the game world with screen-space interfaces, introducing game-specific biases and irrelevant dynamics that hinder world-model training. To address this problem, we introduce GameUI-Taxonomy and G2WEngine, a full-stack framework that formalizes gameplay UI grounding and removal. G2WEngine automatically extracts reusable UI assets from real gameplay videos and synthesizes temporally coherent UI overlays on clean footage. Using this engine, we construct Game2World, comprising 96K synthetic paired videos with precise reconstruction targets and 1,079 in-the-wild clips from 303 games for realistic evaluation. Its asset library contains 5,132 verified UI elements across 21 taxonomy categories, collected from 1,010 representative gameplay frames. Based on Game2World, we propose GameCleaner, a mask-free gameplay UI removal model that combines multimodal semantic understanding with video editing capabilities. Unlike mask-based methods, GameCleaner directly identifies and removes diverse HUD elements while preserving the underlying scene content and temporal dynamics. In a controlled pilot, world models trained on UI-free gameplay improve overall VideoReward by 6.83% over those trained on UI-overlaid data. On UI-removal evaluation, GameCleaner achieves an average AAR of 95.36 on synthetic videos, outperforming the strongest temporal mask baseline by 57.3%, and obtains the best in-the-wild AAR of 80.05 with 99.8 background preservation. These results demonstrate the scalable potential of transforming Internet gameplay videos into high-quality world-model training data. Code, dataset, and model will be available at https://github.com/Dongping-Chen/Game2World.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。