用像素流学会玩3D游戏，像人一样反应智能。
Pixels to Play: A Foundation Model for 3D Gameplay
- 用人类游戏录像+逆动力学模型补全动作，端到端训练
- 单张消费级显卡运行，支持多种游戏的通用玩法
- 适合做队友、NPC或游戏测试助手，可扩展为文本控制
我们提出Pixels2Play-0.1（P2P0.1），一个能以类人行为游玩多种3D游戏的基础模型。针对AI队友、可控非玩家角色、个性化直播主及辅助测试等新兴需求，该模型仅依赖玩家可见的像素流，无需针对每款游戏进行大量定制开发。训练采用行为克隆：结合人工录制的游戏演示与公开未标注视频，通过逆动力学模型推断动作标签。模型采用仅解码器的自回归变压器架构，在单张消费级显卡上保持低延迟。定性结果显示其在简单Roblox和经典MS-DOS游戏中表现良好；消融实验验证了未标注数据的价值，并指出了实现专家级、文本条件控制所需的扩展与评估路径。
原文摘要 · Abstract (English)
We introduce Pixels2Play-0.1 (P2P0.1), a foundation model that learns to play a wide range of 3D video games with recognizable human-like behavior. Motivated by emerging consumer and developer use cases - AI teammates, controllable NPCs, personalized live-streamers, assistive testers - we argue that an agent must rely on the same pixel stream available to players and generalize to new titles with minimal game-specific engineering. P2P0.1 is trained end-to-end with behavior cloning: labeled demonstrations collected from instrumented human game-play are complemented by unlabeled public videos, to which we impute actions via an inverse-dynamics model. A decoder-only transformer with auto-regressive action output handles the large action space while remaining latency-friendly on a single consumer GPU. We report qualitative results showing competent play across simple Roblox and classic MS-DOS titles, ablations on unlabeled data, and outline the scaling and evaluation steps required to reach expert-level, text-conditioned control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。