用多源人体动作数据教会机器人灵活抓取,还能自适应真实环境。
HERMES: Human-to-Robot Embodied Learning from Multi-Source Motion Data for Mobile Dexterous Manipulation
- 统一强化学习框架,融合多源人体手势生成机器人的合理动作。
- 通过深度图像实现端到端仿真到现实迁移,减少真实场景偏差。
- 结合视觉定位与导航模型,让机器人在复杂环境中自主完成操作。
利用人体运动数据赋予机器人多样化的操作能力已成为机器人操作领域的重要方向。然而,将多源人体手部动作转化为适用于具备多指灵巧手的机器人行为仍具挑战性,尤其在高维、复杂的动作空间中。此外,现有方法常难以生成能适应多种环境条件的策略。本文提出HERMES,一种面向移动双臂灵巧操作的人机协同学习框架。首先,HERMES构建统一的强化学习方法,可无缝将多源异构的人体手部动作转换为物理上合理的机器人行为;其次,为缓解仿真到现实的差距,设计了基于深度图像的端到端模拟到现实迁移方法,提升在真实场景中的泛化能力;进一步地,通过引入闭环视角-点(PnP)定位机制增强导航基础模型,实现视觉目标精准对齐,有效衔接自主导航与灵巧操作。大量实验表明,HERMES在多种野外复杂场景中均表现出强泛化能力,成功完成多项复杂的移动双臂灵巧操作任务。
原文摘要 · Abstract (English)
Leveraging human motion data to impart robots with versatile manipulation skills has emerged as a promising paradigm in robotic manipulation. Nevertheless, translating multi-source human hand motions into feasible robot behaviors remains challenging, particularly for robots equipped with multi-fingered dexterous hands characterized by complex, high-dimensional action spaces. Moreover, existing approaches often struggle to produce policies capable of adapting to diverse environmental conditions. In this paper, we introduce HERMES, a human-to-robot learning framework for mobile bimanual dexterous manipulation. First, HERMES formulates a unified reinforcement learning approach capable of seamlessly transforming heterogeneous human hand motions from multiple sources into physically plausible robotic behaviors. Subsequently, to mitigate the sim2real gap, we devise an end-to-end, depth image-based sim2real transfer method for improved generalization to real-world scenarios. Furthermore, to enable autonomous operation in varied and unstructured environments, we augment the navigation foundation model with a closed-loop Perspective-n-Point (PnP) localization mechanism, ensuring precise alignment of visual goals and effectively bridging autonomous navigation and dexterous manipulation. Extensive experimental results demonstrate that HERMES consistently exhibits generalizable behaviors across diverse, in-the-wild scenarios, successfully performing numerous complex mobile bimanual dexterous manipulation tasks. Project Page:https://gemcollector.github.io/HERMES/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。