用行为克隆生成价值函数，实现机器人在线强化学习的高效迭代。
When Life Gives You BC, Make Q-functions: Extracting Q-values from Behavior Cloning for On-Robot Reinforcement Learning
- 从行为克隆策略中提取Q值，通过少量环境交互构建价值估计。
- 结合Q值选择动作，使机器人在1-2小时内达成100%成功率。
- 适合高精度接触类任务，如管道装配和分拣，可快速提升性能。
行为克隆（BC）已成为机器人学习的有效范式，但缺乏收集演示后在线优化的自引导机制。现有离线到在线学习方法常因离线数据与在线学习间的分布差异，导致策略替换已有良好动作。本文提出Q2RL——一种基于行为克隆进行Q值估计与门控的高效离线到在线学习算法。该方法包含两部分：(1) 使用少量环境交互从BC策略中提取Q函数；(2) 通过Q值门控，在BC与强化学习策略间动态切换动作，以生成高质量样本用于在线强化学习。在D4RL和robomimic基准的各类操作任务上，Q2RL在成功率和收敛速度上均优于当前最优基线。该方法足够高效，可在真实机器人上部署，仅需1-2小时在线交互即可学习出鲁棒策略，完成接触密集且高精度的操作任务（如管道装配、分拣），成功率达100%，较原始BC策略提升最高达3.75倍。代码与视频见https://pages.rai-inst.com/q2rl_website/
原文摘要 · Abstract (English)
Behavior Cloning (BC) has emerged as a highly effective paradigm for robot learning. However, BC lacks a self-guided mechanism for online improvement after demonstrations have been collected. Existing offline-to-online learning methods often cause policies to replace previously learned good actions due to a distribution mismatch between offline data and online learning. In this work, we propose Q2RL, Q-Estimation and Q-Gating from BC for Reinforcement Learning, an algorithm for efficient offline-to-online learning. Our method consists of two parts: (1) Q-Estimation extracts a Q-function from a BC policy using a few interaction steps with the environment, followed by online RL with (2) Q-Gating, which switches between BC and RL policy actions based on their respective Q-values to collect samples for RL policy training. Across manipulation tasks from D4RL and robomimic benchmarks, Q2RL outperforms SOTA offline-to-online learning baselines on success rate and time to convergence. Q2RL is efficient enough to be applied in an on-robot RL setting, learning robust policies for contact-rich and high precision manipulation tasks such as pipe assembly and kitting, in 1-2 hours of online interaction, achieving success rates of up to 100% and up to 3.75x improvement against the original BC policy. Code and video are available at https://pages.rai-inst.com/q2rl_website/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。