arXiv:2506.09995cs.CV2025-06NeurIPS被引 20

首个沉浸式第一视角世界模拟器,能精准生成与真实动作对齐的动态视频。

PlayerOne: Egocentric World Simulator

  • 分阶段训练:先用大规模图文对预训练,再用同步动作视频微调。
  • 支持部位级运动控制,实现对肢体动作的精确调节。
  • 可生成长时序一致场景视频,适合虚拟现实与动作分析应用。

我们提出PlayerOne,首个第一视角的真实世界模拟器,支持在生动动态环境中进行沉浸式、无限制探索。给定用户的第一视角场景图像,PlayerOne能准确构建对应世界,并生成严格对齐真实人体动作的第二视角视频。模型采用粗到细训练流程:首先在大规模第一视角文本-视频对上预训练以获得粗粒度理解,随后在从第一/第二视角视频数据集中提取的同步动作-视频数据上微调,使用我们设计的自动构建管道。此外,针对不同组件的重要性差异,设计了部件解耦的动作注入方案,实现对局部运动的精确控制;同时提出联合重建框架,逐步建模4D场景与视频帧,确保长序列视频生成中的场景一致性。实验表明其在多样化人体动作精确控制和复杂场景一致性建模方面具有优异泛化能力。该工作首次探索第一视角真实世界模拟,为世界建模及其多样化应用开辟新方向。

原文摘要 · Abstract (English)

We introduce PlayerOne, the first egocentric realistic world simulator, facilitating immersive and unrestricted exploration within vividly dynamic environments. Given an egocentric scene image from the user, PlayerOne can accurately construct the corresponding world and generate egocentric videos that are strictly aligned with the real scene human motion of the user captured by an exocentric camera. PlayerOne is trained in a coarse-to-fine pipeline that first performs pretraining on large-scale egocentric text-video pairs for coarse-level egocentric understanding, followed by finetuning on synchronous motion-video data extracted from egocentric-exocentric video datasets with our automatic construction pipeline. Besides, considering the varying importance of different components, we design a part-disentangled motion injection scheme, enabling precise control of part-level movements. In addition, we devise a joint reconstruction framework that progressively models both the 4D scene and video frames, ensuring scene consistency in the long-form video generation. Experimental results demonstrate its great generalization ability in precise control of varying human movements and worldconsistent modeling of diverse scenarios. It marks the first endeavor into egocentric real-world simulation and can pave the way for the community to delve into fresh frontiers of world modeling and its diverse applications.

世界建模第一视角动作生成视频模拟

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。