用分层策略+助手操控,让机器人更高效学会精细抓取。
NestDex: Nested Policy Learning with Copilot Assisted Teleoperation for Dexterous Manipulation

- 操作者只控机械臂和切换手部技能,无需手动规划每根手指动作。
- 真实场景实验中,示范收集效率提升,自主策略学习效果显著。
- 适合想快速训练复杂手部操作的机器人研发人员。
精细操作能极大拓展机器人与物理世界的交互能力,但学习此类行为受限于难以获取一致且完整的示范数据。与平行夹持不同,精细任务需操作者在全程协调机械臂运动与高接触频率的手指动作。我们提出NestDex,一种分层策略学习框架,通过学习的手部技能辅助示范采集。操作者仅通过单自由度离合器控制机械臂并切换激活的手部技能,而非直接指定完整手指轨迹。内部手部策略基于最新本体感知历史自适应调整动作,视觉-语言选择器则根据任务阶段激活合适技能。生成的示范用于训练独立的外部视觉-运动策略,部署时无需内层策略。手部动作变分自编码器提供紧凑的手部动作目标,同时保留关节空间的机械臂指令。在真实世界精细操作实验中,NestDex显著提升示范可靠性与效率,实证评估验证了其自主策略学习的有效性。视频演示见项目网站 https://aus.bot/research/nestdex。
原文摘要 · Abstract (English)
Dexterous manipulation promises substantially richer robot interaction with the physical world, but learning these behaviours remains constrained by the difficulty of collecting consistent, complete-task demonstrations. Unlike parallel-jaw manipulation, dexterous tasks require the operator to coordinate arm motion with precise, contact-rich finger behaviour throughout the task. We introduce NestDex, a nested policy-learning framework that reduces this burden by using learned hand skills to assist demonstration collection. The operator controls the arm and regulates the active hand skill through a single-DoF clutch, rather than directly specifying the full finger trajectory. The inner hand policy adapts its motion from the latest proprioceptive history, while a vision-language selector activates the appropriate skill for each task stage. The resulting demonstrations train a separate outer visuomotor policy that controls both the arm and hand without the inner policies at deployment. A hand-action variational autoencoder provides compact hand-action targets while retaining arm commands in joint space. Across real-world dexterous manipulation experiments, NestDex improves demonstration reliability and efficiency, and the resulting empirical evaluations support effective autonomous policy learning. Video Demo are available at project website https://aus.bot/research/nestdex.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。