arXiv:2511.10987cs.RO2025-11中稿 · AAAI

用人类操作视频生成机器人灵巧操作轨迹,无需大量训练数据

Dexterous Manipulation Transfer via Progressive Kinematic-Dynamic Alignment

  • 通过分步对齐人体与机器手的运动学和动力学差异
  • 在不同任务下平均迁移成功率73%,生成流畅且语义正确的动作
  • 适合缺乏数据的灵巧操作研究者快速构建仿真或真实数据集

多指机器人手硬件平台收集操作数据成本高、可扩展性差,导致灵巧操作数据严重匮乏,制约了数据驱动策略的学习。为此,我们提出一种无手型的灵巧操作迁移系统,仅需人类操作视频即可高效生成高质量灵巧操作轨迹,无需大规模训练数据。针对人体手与灵巧手间的多维差异及高自由度协同控制挑战,设计渐进式迁移框架:首先基于运动学匹配建立灵巧手的基础控制信号;随后通过动作空间重缩放与拇指引导初始化训练残差策略,在统一奖励下动态优化接触交互;最后计算腕部控制轨迹以保持操作语义一致性。仅使用人类操作视频,系统可自动配置不同任务的参数,平衡灵巧手、物体类别与任务间的运动学匹配与动态优化。大量实验表明,该框架能自动生成平滑且语义正确的灵巧手操作,忠实还原人类意图,平均迁移成功率73%,具备高效率与强泛化能力,为灵巧操作数据采集提供可复现、可扩展的新方法。

原文摘要 · Abstract (English)

The inherent difficulty and limited scalability of collecting manipulation data using multi-fingered robot hand hardware platforms have resulted in severe data scarcity, impeding research on data-driven dexterous manipulation policy learning. To address this challenge, we present a hand-agnostic manipulation transfer system. It efficiently converts human hand manipulation sequences from demonstration videos into high-quality dexterous manipulation trajectories without requirements of massive training data. To tackle the multi-dimensional disparities between human hands and dexterous hands, as well as the challenges posed by high-degree-of-freedom coordinated control of dexterous hands, we design a progressive transfer framework: first, we establish primary control signals for dexterous hands based on kinematic matching; subsequently, we train residual policies with action space rescaling and thumb-guided initialization to dynamically optimize contact interactions under unified rewards; finally, we compute wrist control trajectories with the objective of preserving operational semantics. Using only human hand manipulation videos, our system automatically configures system parameters for different tasks, balancing kinematic matching and dynamic optimization across dexterous hands, object categories, and tasks. Extensive experimental results demonstrate that our framework can automatically generate smooth and semantically correct dexterous hand manipulation that faithfully reproduces human intentions, achieving high efficiency and strong generalizability with an average transfer success rate of 73%, providing an easily implementable and scalable method for collecting robot dexterous manipulation data.

灵巧操作动作迁移视频生成机器人学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。