arXiv:2608.15917cs.ROcs.AI2026-08

用虚拟现实模拟人类操作,高效收集多指灵巧操作数据,提升机器人实操性能。

Pre-training Visual Dexterity in Simulation

论文配图:Pre-training Visual Dexterity in Simulation
图 1 · 摘自论文原文
  • 通过VR让操作员在仿真中操控虚拟物体,直接获取与机器人匹配的动作轨迹。
  • 仅用1-2小时真实演示微调,即在56自由度双臂系统上实现超越从零训练的性能。
  • 适合研究灵巧操作、数据效率和仿真预训练的团队,尤其关注低成本数据采集。

大规模预训练已显著提升机器人策略微调的数据效率,但主要依赖简单夹爪的实物数据集。多指灵巧手仍面临数据稀缺问题:真实遥操作成本高,而人类手部视频为非本体动作,需经有损姿态估计与重定向。本文提出仿真灵巧操作预训练框架SPD,完全基于仿真数据。操作员佩戴VR头显在虚拟环境中操控物体,实现本体化轨迹采集,无需真实机器人。五名操作员一周内收集75小时多任务灵巧操作数据,用于在因果Transformer上进行序列建模预训练。通过在56自由度双臂灵巧系统上仅用1-2小时物理示范进行微调,验证了该方法的有效性。结果表明,相比从零训练行为克隆策略,其表现更优,证明仿真遥操作是真实世界灵巧操作的可行预训练来源。进一步的消融实验分析了历史条件与短动作块对反应式控制的贡献。

原文摘要 · Abstract (English)

Large-scale pre-training has made robot policy fine-tuning increasingly data-efficient, but this progress has largely been driven by datasets and embodiments built around simple parallel-jaw grippers. Dexterous, multi-fingered hands remain comparatively data-starved because real teleoperation is costly to scale, while human hand video is off-embodiment and requires lossy pose estimation and retargeting. We introduce Simulation Pre-training for Dexterity (SPD), a pre-training framework for dexterous manipulation that uses data entirely collected in simulation. In SPD, humans manipulate virtual objects inside a VR headset, enabling on-embodiment trajectories and robot-free collection. With the help of five operators, we collect 75 hours of multi-task dexterous manipulation over one week, and use it to pre-train a causal transformer on a sequence modeling objective. We study the benefits of simulation pre-training on real-world tasks by fine-tuning on 1-2 hours of physical demonstrations on a 56-DoF bimanual dexterous setup. We find that our approach outperforms training behavior cloning policies from scratch, showing that simulation teleoperation is a viable pre-training source for real-world dexterous manipulation. We perform ablation studies, measuring the benefits of history conditioning and short action chunks for reactive control.

灵巧操作仿真预训练虚拟现实数据效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。