用手势教学采集带触觉反馈的精细操作数据,提升灵巧抓取成功率。
KineDex: Learning Tactile-Informed Visuomotor Policies via Kinesthetic Teaching for Dexterous Manipulation
- 通过手把手教学直接传递动作,结合触觉反馈采集真实物理交互数据。
- 在挤牙膏等高难度任务中平均成功率74.4%,比无力控版本提升57.7%。
- 数据采集速度是遥操作的两倍以上,且成功率接近100%,适合实际应用。
获取富含细粒度触觉信息的示范数据对灵巧操作至关重要,尤其在需要精确力控和物理交互的接触密集型任务中。现有方法多依赖遥操作或视频重定向,常因运动学不匹配和缺乏实时触觉反馈,难以获得高质量触觉数据。为此,我们提出KineDex,一种手把手式本体感觉教学范式,将操作者动作直接映射到灵巧手,实现带有准确触觉反馈的物理化示范采集。为解决人类手部遮挡问题,采用图像修复技术预处理视觉观测。基于这些示范,我们使用增强触觉输入训练视觉-运动策略,并在部署时引入力控以实现精准接触操作。我们在一系列挑战性接触密集型任务上评估KineDex,包括挤牙膏等极难任务,要求多指协调与稳定力调节。结果显示,其平均成功率达74.4%,较无力控变体提升57.7%。与遥操作对比实验及用户研究进一步验证了KineDex在数据采集效率和易用性上的优势:在两个不同难度任务中,其数据采集速度超过遥操作两倍,且成功率接近100%,而遥操作成功率低于50%。
原文摘要 · Abstract (English)
Collecting demonstrations enriched with fine-grained tactile information is critical for dexterous manipulation, particularly in contact-rich tasks that require precise force control and physical interaction. While prior works primarily focus on teleoperation or video-based retargeting, they often suffer from kinematic mismatches and the absence of real-time tactile feedback, hindering the acquisition of high-fidelity tactile data. To mitigate this issue, we propose KineDex, a hand-over-hand kinesthetic teaching paradigm in which the operator's motion is directly transferred to the dexterous hand, enabling the collection of physically grounded demonstrations enriched with accurate tactile feedback. To resolve occlusions from human hand, we apply inpainting technique to preprocess the visual observations. Based on these demonstrations, we then train a visuomotor policy using tactile-augmented inputs and implement force control during deployment for precise contact-rich manipulation. We evaluate KineDex on a suite of challenging contact-rich manipulation tasks, including particularly difficult scenarios such as squeezing toothpaste onto a toothbrush, which require precise multi-finger coordination and stable force regulation. Across these tasks, KineDex achieves an average success rate of 74.4%, representing a 57.7% improvement over the variant without force control. Comparative experiments with teleoperation and user studies further validate the advantages of KineDex in data collection efficiency and operability. Specifically, KineDex collects data over twice as fast as teleoperation across two tasks of varying difficulty, while maintaining a near-100% success rate, compared to under 50% for teleoperation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。