用物理模型让机器人从生成视频中学会真实操作。
Robot Learning from a Physical World Model
- 结合视频生成与物理重建，将视觉动作转为物理可执行指令。
- 零样本泛化下比现有方法显著提升操作精度。
- 适合想免实机数据训练的机器人研究者使用。
我们提出 PhysWorld 框架，实现机器人通过物理世界建模从视频生成中学习。当前视频生成模型能根据语言指令和图像合成逼真视觉演示，但直接将像素运动映射到机器人会忽略物理规律，导致操作不准。PhysWorld 通过将视频生成与物理世界重建结合，在给定单张图像和任务指令时，生成任务相关的视频并从中重建底层物理世界；再利用基于物理世界的对象中心残差强化学习，将生成视频中的运动转化为物理准确的动作。该协同机制将隐式视觉引导转化为可执行的机器人轨迹，无需真实机器人数据收集，实现零样本泛化机器人操作。在多个真实任务上的实验表明，PhysWorld 显著优于以往方法，大幅提高操作准确性。
原文摘要 · Abstract (English)
We introduce PhysWorld, a framework that enables robot learning from video generation through physical world modeling. Recent video generation models can synthesize photorealistic visual demonstrations from language commands and images, offering a powerful yet underexplored source of training signals for robotics. However, directly retargeting pixel motions from generated videos to robots neglects physics, often resulting in inaccurate manipulations. PhysWorld addresses this limitation by coupling video generation with physical world reconstruction. Given a single image and a task command, our method generates task-conditioned videos and reconstructs the underlying physical world from the videos, and the generated video motions are grounded into physically accurate actions through object-centric residual reinforcement learning with the physical world model. This synergy transforms implicit visual guidance into physically executable robotic trajectories, eliminating the need for real robot data collection and enabling zero-shot generalizable robotic manipulation. Experiments on diverse real-world tasks demonstrate that PhysWorld substantially improves manipulation accuracy compared to previous approaches. Visit \href{https://pointscoder.github.io/PhysWorld_Web/}{the project webpage} for details.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。