从单目视频生成稳定可迁移的灵巧操作轨迹
C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Video

- 用物体坐标系聚合帧间噪声,构建稳定接触表示
- 实测端到端成功率57.78%(DexYCB),超基线3倍以上
- 适合需高保真接触交互的灵巧机器人任务
高质量灵巧操作示范采集成本高,而单目人体视频提供了可扩展的多样化操作数据源。然而,将此类示范迁移到灵巧机器人仍具挑战:单目手物交互重建常产生时间不稳的接触和物理上不合理的交互,传统迁移方法难以保持任务相关的接触和局部交互几何。我们提出C2Dex,基于共享交互表示的视频到灵巧操作框架:通过在标准物体空间聚合帧级噪声观测,恢复稳定的物体侧接触。这些稳定接触起双重作用:作为轨迹级约束,引导重建获得时序一致且物理合理的手物交互轨迹;作为显式迁移目标,通过拉普拉斯交互优化在不同手部形态间保持局部手物几何,残差强化学习在仿真中精炼轨迹。在DexYCB和TACO上的实验表明,C2Dex实现端到端轨迹成功率分别为57.78%和26.67%,显著优于最强基线(17.78%和10.00%)。真实机器人重放实验进一步验证了其在多样接触密集操作任务中的物理可行性。
原文摘要 · Abstract (English)
High-quality demonstrations for dexterous robot manipulation are costly and difficult to collect, whereas monocular human videos provide a scalable source of diverse manipulation behaviors. However, transferring such demonstrations to dexterous robots remains challenging: monocular hand-object interaction (HOI) reconstruction often produces temporally unstable contacts and physically implausible interactions, while conventional retargeting methods struggle to preserve task-relevant contacts and local interaction geometry across different hand embodiments. We present C2Dex, a video-to-dexterous-manipulation framework built around a shared interaction representation: stable object-side contacts recovered by aggregating noisy frame-wise observations in the canonical object space. These stable contacts serve a dual role: as trajectory-level constraints that guide reconstruction toward temporally coherent and physically plausible human HOI trajectories, and as explicit transfer targets for the dexterous hand, where Laplacian interaction optimization preserves the local hand-object geometry across embodiments and residual reinforcement learning refines the trajectory in simulation. Experiments on DexYCB and TACO show that C2Dex achieves end-to-end trajectory success rates of 57.78% and 26.67%, respectively, substantially outperforming the strongest baselines (17.78% and 10.00%) under identical evaluation criteria. Real-robot replay experiments further demonstrate physical feasibility across diverse contact-rich manipulation tasks. Project page: https://k-jie.github.io/C2Dex/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。