arXiv:2604.13015cs.RO2026-04被引 6

通过触觉梦境增强学习,让机器人手更灵巧地完成复杂操作。

Learning Versatile Humanoid Manipulation with Touch Dreaming

论文配图:Learning Versatile Humanoid Manipulation with Touch Dreaming
图 1 · 摘自论文原文
  • 用强化学习构建稳定下肢控制器,支持全身协同操作。
  • 触觉梦境训练使真实任务成功率提升90.9%,触觉潜空间预测更有效。
  • 适合研究灵巧操作、触觉感知与具身智能的开发者和科研人员。

人形机器人有望实现通用辅助,但现实中的全身运动-操作仍面临挑战,需兼顾全身稳定、末端执行器灵巧性以及频繁接触变化下的感知交互。本文研究高接触密度的人形机器人运动-操作任务。首先设计基于强化学习的下肢控制器,作为复杂操作中的稳定性基础。在此基础上,构建结合灵巧手与触觉传感的虚拟现实数据采集系统。提出触觉梦境人形变换器(HTD),一种多模态编码-解码Transformer,将触觉作为核心模态,融合多视角视觉与本体感觉。HTD采用单阶段行为克隆训练,结合触觉梦境:除预测动作片段外,还预测未来手部关节力与触觉潜变量,触觉潜变量目标由指数移动平均目标编码器提供,无需独立触觉预训练。该机制促使策略学习接触感知表征。在五个真实世界高接触任务中,HTD相较更强基线平均成功率提升90.9%。消融实验表明,潜空间触觉预测比原始触觉预测更优,成功率相对提升30%。结果表明,触觉梦境增强的学习系统可实现真实世界中高灵巧度、多功能的人形操作。更多信息与开源资源见 humanoid-touch-dream.github.io。

原文摘要 · Abstract (English)

Humanoid robots promise general-purpose assistance, yet real-world humanoid loco-manipulation remains challenging because it requires whole-body stability, end-effector dexterity, and contact-aware interaction under frequent contact changes. In this work, we study dexterous, contact-rich humanoid loco-manipulation. We first develop an RL-based lower-body controller that serves as the stability backbone for whole-body execution during complex manipulation. Building on this controller, we develop a VR-based whole-body humanoid data collection system that integrates dexterous hands and tactile sensing for contact-rich manipulation. We then propose Humanoid Transformer with Touch Dreaming (HTD), a multimodal encoder-decoder Transformer that models touch as a core modality alongside multi-view vision and proprioception. HTD is trained in a single stage with behavioral cloning augmented by touch dreaming: in addition to predicting action chunks, the policy predicts future hand-joint forces and future tactile latents, with tactile-latent targets provided by an exponential moving average target encoder without requiring a separate tactile pretraining stage. This encourages the policy to learn contact-aware representations for dexterous manipulation. Across five real-world contact-rich tasks, HTD achieves a 90.9% relative improvement in average success rate over the stronger baseline for each task. Ablation results further show that latent-space tactile prediction is more effective than raw tactile prediction, yielding a 30% relative gain in success rate. These results demonstrate that our touch-dreaming-enhanced learning system enables versatile, high-dexterity humanoid manipulation in the real world. More information and open-source materials are available at humanoid-touch-dream.github.io.

人形机器人触觉感知强化学习灵巧操作

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。