用生成模型替代真实机器人,实现高效远程操控数据采集。
RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation

- 用动作条件世界模型生成视角视频,解耦操作与硬件绑定。
- 单张参考图+手部动作流,可实时生成40+帧/秒的高清视频。
- 生成数据可直接用于机器人模仿学习,支持跨平台零样本迁移。
机器人学习需要海量多样轨迹数据,但物理远程操控受限于操作员时间与特定设备。本文提出数字远程操控新范式,用生成式世界模型替代真实机器人,将操作员的手部姿态流输入机器人中心的生成模型,从单张参考图像合成高保真第一人称视频。记录的姿态流作为与具体硬件无关的动作标签,可通过标准重定向应用于任意目标机器人,生成完整状态-动作轨迹,实现不依赖物理设备的模仿学习。我们构建了RynnWorld-Teleop系统,融合深度感知骨骼条件、视频扩散变换器的渐进式人机训练以及流式自回归蒸馏。该流程压缩为单次推理,可在单个H100 GPU上实现40+ FPS的实时交互生成。仅使用生成数据训练的策略在多种灵巧双臂任务中实现了有效的零样本仿真到现实迁移。此外,将生成数据加入真实数据集后,成功率持续提升,证明RynnWorld-Teleop是下一代机器人智能的高保真、可扩展数据引擎。
原文摘要 · Abstract (English)
Scaling robot learning requires massive, diverse trajectory data, yet collection is currently bottlenecked by physical teleoperation, where every demonstration binds operator time to specific hardware and workspaces. We introduce digital teleoperation, a paradigm that decouples data collection from physical constraints by replacing the real robot with a generative world model. In this framework, an operator's hand-pose stream drives a robot-centric generative world model to synthesize high-fidelity egocentric videos from a single reference image. The recorded pose stream serves as an embodiment-agnostic action label transferable to any target robot via standard retargeting, yielding complete state-action trajectories for imitation learning independent of physical hardware. We instantiate this paradigm in RynnWorld-Teleop, a system that integrates depth-aware skeletal conditioning, progressive human-to-robot training on a video Diffusion Transformer, and streaming autoregressive distillation. This pipeline compresses the generative process into a single-pass inference, enabling 40+ FPS, real-time interactive generation on a single H100 GPU. Policies trained exclusively on RynnWorld-Teleop-generated data achieve effective zero-shot Sim2Real transfer across dexterous and diverse bimanual tasks. Moreover, augmenting real-world datasets with our digitally teleoperated data consistently improves success rates, demonstrating that RynnWorld-Teleop serves as a high-fidelity, scalable data engine for the next generation of robotic agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。