EgoSim生成连贯的视角交互视频,支持3D场景持续更新。
EgoSim: Egocentric World Simulator for Embodied Interaction Generation

- 用可更新的3D世界状态建模,实现视角变化下的空间一致性。
- 在复杂场景和真实操作中,视觉质量与一致性显著优于现有方法。
- 适合机器人操作、虚拟人交互等需要长期动态模拟的研究者。
我们提出EgoSim,一个闭环的自我中心世界模拟器,能生成空间一致的交互视频,并持续更新底层3D场景状态以支持连续模拟。现有自我中心模拟器或缺乏显式3D定位,导致视角变化时结构漂移;或将场景视为静态,无法在多阶段交互中更新世界状态。EgoSim通过将3D场景建模为可更新的世界状态,解决了上述问题。我们采用几何-动作感知的观察模拟模型生成具身交互,借助交互感知的状态更新模块保证空间一致性。为克服高质量场景-交互配对数据稀缺的瓶颈,我们设计了可扩展的数据提取流程,从真实世界的单目自我中心视频中提取静态点云、相机轨迹和具身动作。此外,我们提出了EgoCap系统,使用未标定智能手机低成本采集真实数据。大量实验表明,EgoSim在视觉质量、空间一致性及复杂场景和真实灵巧操作的泛化能力上均显著优于现有方法,且支持跨具身迁移至机器人操作。代码与数据集即将开源,项目主页为 egosimulator.github.io。
原文摘要 · Abstract (English)
We introduce EgoSim, a closed-loop egocentric world simulator that generates spatially consistent interaction videos and persistently updates the underlying 3D scene state for continuous simulation. Existing egocentric simulators either lack explicit 3D grounding, causing structural drift under viewpoint changes, or treat the scene as static, failing to update world states across multi-stage interactions. EgoSim addresses both limitations by modeling 3D scenes as updatable world states. We generate embodiment interactions via a Geometry-action-aware Observation Simulation model, with spatial consistency from an Interaction-aware State Updating module. To overcome the critical data bottleneck posed by the difficulty in acquiring densely aligned scene-interaction training pairs, we design a scalable pipeline that extracts static point clouds, camera trajectories, and embodiment actions from in-the-wild large-scale monocular egocentric videos. We further introduce EgoCap, a capture system that enables low-cost real-world data collection with uncalibrated smartphones. Extensive experiments demonstrate that EgoSim significantly outperforms existing methods in terms of visual quality, spatial consistency, and generalization to complex scenes and in-the-wild dexterous interactions, while supporting cross-embodiment transfer to robotic manipulation. Codes and datasets will be open soon. The project page is at egosimulator.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。