arXiv:2502.02705cs.ROcs.LG2025-02ICLR被引 23

用仿真引导的微调,让机器人用极少真实数据快速学会精细操作。

Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuning

  • 用仿真中学习的价值函数指导真实世界探索,提升采样效率。
  • 在5个精细操作任务中,所需真实样本减少一个数量级,成功率显著提升。
  • 适合需要高精度控制的机器人实际应用,尤其适用于数据稀缺场景。

机器人学习需要大量高质量数据以实现泛化,但真实世界数据收集成本高昂。物理模拟器可低成本生成覆盖广泛状态、动作和环境的数据集,但其对现实的建模存在根本性偏差,导致直接零样本从仿真到现实迁移困难,尤其在需要精确力控的任务中。因此,利用少量真实数据对策略进行微调成为规模化机器人学习的可行路径。然而,现有强化学习微调框架依赖通用、无结构的探索策略,效率过低,难以实用。本文提出仿真引导微调(SGFT)框架,通过从物理模拟器中提取结构先验,显著加速真实世界适应。具体而言,利用模拟中学习的价值函数引导真实世界探索。我们在五个真实世界的灵巧操作任务中验证该方法,零样本仿真到现实迁移均失败。结果表明,本框架显著优于基线微调方法,所需真实样本减少达一个数量级,在先前方法完全失败的复杂任务上仍能成功。最后,我们提供了该范式的理论依据,说明SGFT如何在巨大仿真-现实差异下快速学习高性能策略。项目页面:https://weirdlabuw.github.io/sgft/

原文摘要 · Abstract (English)

Robot learning requires a considerable amount of high-quality data to realize the promise of generalization. However, large data sets are costly to collect in the real world. Physics simulators can cheaply generate vast data sets with broad coverage over states, actions, and environments. However, physics engines are fundamentally misspecified approximations to reality. This makes direct zero-shot transfer from simulation to reality challenging, especially in tasks where precise and force-sensitive manipulation is necessary. Thus, fine-tuning these policies with small real-world data sets is an appealing pathway for scaling robot learning. However, current reinforcement learning fine-tuning frameworks leverage general, unstructured exploration strategies which are too inefficient to make real-world adaptation practical. This paper introduces the Simulation-Guided Fine-tuning (SGFT) framework, which demonstrates how to extract structural priors from physics simulators to substantially accelerate real-world adaptation. Specifically, our approach uses a value function learned in simulation to guide real-world exploration. We demonstrate this approach across five real-world dexterous manipulation tasks where zero-shot sim-to-real transfer fails. We further demonstrate our framework substantially outperforms baseline fine-tuning methods, requiring up to an order of magnitude fewer real-world samples and succeeding at difficult tasks where prior approaches fail entirely. Last but not least, we provide theoretical justification for this new paradigm which underpins how SGFT can rapidly learn high-performance policies in the face of large sim-to-real dynamics gaps. Project webpage: https://weirdlabuw.github.io/sgft/{weirdlabuw.github.io/sgft}

机器人学习仿真迁移微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。