arXiv:2511.01177cs.RO2025-11中稿 · IROS 2026, Project…被引 11

用粒子模型统一人手与机器人的物理交互,实现跨形态操控通用学习。

Scaling Cross-Embodiment World Models for Dexterous Manipulation

  • 将手部建模为3D粒子集合,通过位移场定义动作,抽象出共性几何结构。
  • 在多种仿真与真实数据上训练后,模型可泛化至未见手型并有效控制不同机器人。
  • 适合研究跨形态机器人、通用操控系统及物理交互建模的开发者参考。

跨形态学习旨在构建能从多样形态中学习并操作的通用机器人,但运动学与动作空间差异阻碍了数据共享与控制迁移。本文提出:尽管存在差异,物理交互所引发的结构可在共享几何空间中建模,从而让世界模型成为学习与控制的统一接口。为此,将人手与机器人手表示为3D粒子集合,动作定义为末端粒子的位移场,该表示方式剥离了特定构型的关节空间,保留了与物理交互相关的几何与运动信息。我们在多种仿真机器人手和真实人手的随机交互数据上训练了一个基于图的世界模型,并将其与模型预测控制结合,部署于新硬件。实验表明:增加训练形态多样性可提升对未知手型的泛化能力;合理融合仿真与真实数据优于单一来源;同一模型可有效控制具有不同运动学与自由度的机器人手。结果表明,基于粒子的世界模型可作为异构形态间的学习与控制共享接口。

原文摘要 · Abstract (English)

Cross-embodiment learning seeks to build generalist robots that learn from and operate across diverse morphologies, but differences in kinematics and action spaces hinder data sharing and control transfer. We ask: What structure can be shared across embodiments despite these differences? We argue that the physical interactions they induce can be modeled in a shared geometric space, allowing world models to provide a common interface for learning and control. To realize this idea, we represent human and robot hands as sets of 3D particles and define actions as end-effector particle displacement fields. This representation abstracts away embodiment-specific joint spaces while preserving the geometry and motion relevant to physical interaction. We train a graph-based world model on random interaction data from diverse simulated robot hands and real human hands, and integrate it with model-predictive control for deployment on new hardware. Experiments on rigid and deformable manipulation reveal three findings: increasing the diversity of training embodiments improves generalization to unseen hands; appropriately combining simulated and real-world data outperforms either source alone; and the same learned model enables effective control on robotic hands with distinct kinematics and degrees of freedom. These results position particle-based world models as a shared interface for learning from and for heterogeneous embodiments.

机器人操控世界模型跨形态学习粒子建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。