让机器人通过失败自我改进，不依赖人类示范。
Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning
- 用小规模Q函数引导大模型决策，实现价值驱动的动作选择。
- 仅更新Q函数，十轮迭代后成功率从93%升至99%。
- 真实机器人任务中纯靠自身部署数据提升，无需人工干预。
行为克隆（BC）在机器人操作中取得显著进展，但无法自我改进：失败的策略无法从中学习，除非有额外的人类示范。强化学习微调虽可实现自进化，但难以扩展到现代机器人策略所依赖的数十亿参数大模型。我们提出Q-Planning，为大型视觉-运动BC策略配备小型离策略Q函数。由于Q函数估计的是价值而非模仿动作，它可使用与BC相同的成功示范进行训练，并在后续吸收成功和失败的部署轨迹，这是BC不具备的不对称性。我们利用这一特性，在推理时实现价值引导的动作选择（单步Q加权平均），并在线实现仅更新Q函数的自改进，保持BC权重不变。在LIBERO和双臂RoboTwin上，经过十轮自改进，所有测试基准分数均提升（LIBERO-10从93%到99%，RoboTwin从83.8%到91.4%），且在接近天花板的场景中缩短了成功轨迹。在两个接触丰富的双臂真实机器人任务中，相同循环（冻结BC，无额外人工）仅基于自身部署轨迹实现提升：叠杯子从40%到90%，插钱包从25%到80%（五轮），而仅用成功轨迹进行SFT则停滞在55%和30%。在相同在线预算下，相比Best-of-N、过滤式SFT、IBRL、DSRL和DAWR，Q-Planning是唯一能稳定从失败中学习而不需辅助动作器的方法。
原文摘要 · Abstract (English)
Behaviour Cloning (BC) has driven remarkable progress in robot manipulation, yet it is fundamentally limited by its inability to self-improve: a policy that fails cannot learn from that failure without additional human demonstrations. Reinforcement Learning fine-tuning offers a path to self-improvement but has proven difficult to scale to the multi-billion-parameter models underpinning modern robot policies. We propose Q-Planning, which equips a large visuomotor BC policy with a small off-policy Q-function. Because a Q-function estimates value rather than imitates actions, it can be trained on the same successful demonstrations as the BC policy and later absorb both successful and failed deployment rollouts, an asymmetry BC does not have. We exploit this asymmetry to enable value-guided action selection at inference (a single-step Q-weighted average over BC draws) and online self-improvement that fine-tunes only the Q-function, leaving the BC weights untouched. On LIBERO and bimanual RoboTwin, ten iterations of self-improvement lift every benchmark score we tested (LIBERO-10 93% to 99%, RoboTwin 83.8% to 91.4%) and shorten successful episodes on the near-ceiling suites (LIBERO-Object, LIBERO-Goal). On two contact-rich bimanual real-robot tasks, the same loop (BC frozen, no human intervention) improves purely from its own deployment rollouts: stack-cups 40% to 90% and insert-wallet 25% to 80% in five iterations, whereas SFT on successful rollouts alone stalls at 55% and 30%. Under an identical online budget Q-Planning is the only method, among Best-of-N, filtered SFT, IBRL, DSRL, and DAWR, that improves stably from failures without training an auxiliary actor.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。