无需视觉,仅靠触觉实现真实机械手的盲抓取。
Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning

- 用真实到仿真再回到真实的触觉校准,缩小模拟与现实差距。
- 在10种已知和10种未知物体上达成27%的抓取成功率。
- 适合做无视觉、纯触觉控制的机器人抓取研究者参考。
无视觉的灵巧手抓取是关键操作能力。然而,由于触觉模拟到现实的差距以及稀疏触觉信号表达力有限,学习真实机器人上的纯触觉策略仍具挑战。本文提出一种可部署于物理多指机器人手的纯触觉盲抓取框架。首先,设计了Real2Sim触觉校准流程,构建能复现真实触觉信号的接触校准数字孪生仿真器。其次,通过自监督预训练引入传感器几何先验,提升稀疏触觉观测的表达能力。第三,为提升对未知物体的泛化能力,在校准仿真器中训练物体特定强化学习专家,并将其成功抓取轨迹聚合为触觉条件扩散策略。在配备分布式触觉传感的LEAP Hand上评估,对10种已知和10种未知物体进行测试,部署策略在所有20个物体上实现27%的真实世界抓取成功率,无需真实抓取演示或视觉输入。仿真消融实验表明,布局感知触觉预训练提升抓取性能;传感级评估确认Real2Sim校准显著提高仿真与硬件间触觉接触事件的一致性。结果表明,接触事件校准、几何感知触觉表征学习与扩散策略聚合共同构成一条有效的纯触觉盲抓取路径。
原文摘要 · Abstract (English)
Blind grasping with a dexterous hand is a crucial manipulation capability. Nevertheless, learning such tactile-only policies for real robots remains challenging due to the tactile sim-to-real gap and the limited expressiveness of sparse tactile signals. To bridge this gap, we propose a framework for tactile-only blind grasping that is deployable on a physical multi-fingered robotic hand. Our approach combines three key components. First, we introduce a Real2Sim tactile calibration pipeline that constructs a contact-calibrated digital-twin simulator capable of reproducing real tactile signals. Second, we improve the expressiveness of sparse tactile observations using a layout-aware tactile encoder, which incorporates sensor-geometry priors through self-supervised pretraining. Third, to improve generalization to unseen objects, we train object-specific reinforcement-learning experts in the calibrated simulator and aggregate their successful grasp trajectories into a tactile-conditioned Diffusion Policy. We evaluate our method on a physical LEAP Hand equipped with distributed tactile sensing across 10 seen and 10 unseen objects. The deployed policy achieves a 27\% real-world grasp success rate across all 20 objects, without real-world grasping demonstrations or visual input. Simulation ablations show that layout-aware tactile pretraining improves grasping performance, while sensing-level evaluations confirm that Real2Sim calibration increases the consistency of tactile contact events between simulation and hardware. Together, these results suggest that contact-event calibration, geometry-aware tactile representation learning, and diffusion-based policy aggregation provide an effective path toward tactile-only blind grasping on real dexterous robotic hands. Project page:Dex-Blind-Grasp.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。