让机器人快速学习人类干预,同时自我优化,30分钟达成99%成功率。
Max-Q Selective Imitation for Human-in-the-Loop Online Robot Learning

- 用蒙特卡洛块级评价直接奖励干预动作,避免策略更新稀释干预效果。
- 通过最大Q选择性模仿,自动在学人和自改进间切换,减少分布偏移。
- 实测20次演示下30分钟达99%成功,比现有方法快近10倍,适合实时交互场景。
真实机器人的人机协同在线强化学习需快速吸收人类干预,同时持续超越人类初始表现。本文提出一种基于双组件的训练方法:首先,使用MC Q-chunk评论家对回放缓冲区中的蒙特卡洛回报进行块级动作值回归,实现样本平均行为策略评估,使干预轨迹获得直接奖励而非被当前策略TD备份稀释;其次,采用max-Q选择性模仿,在每个状态上以硬胜者通吃规则,模仿当前策略动作与缓冲区样本中Q值更高的那个动作。该规则自动在学习干预与自改进之间切换:当自主策略更强时,目标与策略分布对齐,降低策略-目标样本差距,缓解执行时分布偏移。实际中通过标准评论家集成均值评分候选动作,降低比较噪声,无需软化目标或引入分数差阈值。在含20次示范的真实USB插拔任务中,ACT QChunk-MCBC在30分钟内达到99%成功率,而HIL-SERL需约5小时收敛。仿真环境下,针对Peg Insertion和Square任务,ACT/Flow Q-chunk变体在约半小时有效训练后达成≥96%成功率,优于HIL-SERL、EXPO和E2HiL在成功率-时间前沿的表现。
原文摘要 · Abstract (English)
Human-in-the-loop (HIL) online reinforcement learning for real robots must absorb human interventions quickly while continuing to improve beyond the human prior. We present a training method for this setting based on two components. First, an \emph{MC Q-chunk} critic regresses chunk-level action values onto Monte Carlo returns from the replay buffer, performing sample-average (behavior) policy evaluation so that intervention trajectories are credited directly rather than diluted by current-policy TD backups. Second, \emph{max-Q selective imitation} updates the actor by imitating, at each state, the higher-$Q$ action between the current policy action and a buffer sample under a hard winner-take-all rule. This rule automatically switches between learning from interventions and on-policy self-improvement: when the autonomous policy is stronger, targets align with the policy distribution, reducing the policy--target-sample gap that otherwise induces execution-time distribution shift. In practice we score candidates with a standard critic ensemble mean to reduce comparison noise, without softening targets or introducing score-gap thresholds. On a real USB pick-and-insertion task with 20 demonstrations, ACT QChunk-MCBC attains 99\% success within 30 minutes of HIL training, whereas HIL-SERL requires about 5 hours to converge. In simulation on Peg Insertion and Square, ACT/Flow Q-chunk variants similarly reach $\ge$96\% success within roughly half an hour of effective training, outperforming HIL-SERL, EXPO, and E2HiL on the success--time frontier.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。