将单视角人手演示转为可执行的物理验证机器人轨迹
DemoBridge: A Simulation-in-the-Loop Toolkit for Single-View Human Demonstration Retargeting

- 用全局优化的碰撞感知规划器,联合考虑抓握姿态与全臂碰撞
- 在物理模拟闭环中验证并回溯失败路径,确保动作可执行且稳定
- 支持自动抓取时机推断,适用于机器人操控与策略学习
我们提出DemoBridge,一个将单视角RGB立体视频中的人手演示转化为可执行、物理验证的机器人臂轨迹的工具包。跨身体形态的重定向极具挑战:机器人臂具有长而复杂的结构,其连杆占据远大于人手的碰撞体积。仅对末端位姿求解逆运动学常无无碰撞解,且轨迹需在每个航点都满足此条件。单视角引入噪声,导致示范参考不准确。DemoBridge的核心是一个单次优化的碰撞感知规划器,它联合优化整个关节轨迹,同时考虑替代抓握姿态、全臂与被抓物体的碰撞以及对示范路径的保真度。物理模拟器在循环中运行,逐阶段验证并失败时回溯,使无法直接复现的示范被重新规划而非丢弃。生成的动作序列动态稳定且忠实于原始操作,还可作为策略学习的即用仿真轨迹。抓取时机自动推断,感知后端、机器人和流程模块均可配置替换。我们在三个真实示范任务上评估了全管道重定向,并在受控合成基准上测试了规划器。代码已开源。
原文摘要 · Abstract (English)
We present DemoBridge, an toolkit that turns a single-view RGB stereo recording of a human hand demonstration into an executable, physics-validated robot-arm trajectory. Retargeting across the embodiment gap is hard. A robot arm reaches a target with a long, articulated body whose links carry far more collision volume than a hand. Solving inverse kinematics for the mapped end-effector pose often yields no collision-free solution, and a trajectory imposes this at every waypoint. A single view adds noise, leaving the demonstrated reference inaccurate. At the core of DemoBridge is a single collision-aware planner. It optimizes the whole joint trajectory at once, reasoning jointly over alternative grasp poses, whole-arm and grasped-object collision, and fidelity to the demonstrated path. A physics simulator runs in the loop. It validates each phase as it is produced and backtracks on failure, so a demonstration that cannot be reproduced as given is re-planned rather than discarded. The resulting action sequence is dynamically stable and faithful to the demonstrated manipulation. It also doubles as a ready-to-use simulation rollout for policy learning. Grasp timing is inferred automatically, and the perception backends, robot, and pipeline stages are swappable from configuration. We evaluate whole-pipeline retargeting on three real-demonstration tasks and the planner on a controlled synthetic benchmark. Our code is available at https://gitlab.kuleuven.be/u0123974/demo-bridge/ .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。