用演示引导的强化学习，让多任务机器人训练更快更稳。
GPU-Parallel Multi-Task Reinforcement Learning with Demonstration Guided Policy Optimization
- 用演示数据指导策略优化，提升训练效率。
- 在稀疏奖励下仍实现稳定收敛，性能超越现有方法。
- 适合需要多任务并行训练的机器人研究者。
大规模GPU并行强化学习已改变机器人仿真中的可训练规模，但多数系统仍为每项任务训练单一专用策略。本文提出一种将结构化操作任务族转化为GPU并行多任务强化学习基准的构建方法，并基于LIBERO资源与Isaac Lab中的任务谓词实现了MT-Libero基准。该基准支持异构任务集的并行强化学习，具备并行渲染、物理随机化以及状态输入或视觉输入策略。为应对稀疏成功信号和有限先验数据，本文进一步提出DGPO，一种基于演示的在线策略优化方法，结合重要性加权PPO与自适应行为克隆，可灵活调整对演示任务分布的偏好。DGPO在保持PPO稳定性与在线改进优势的同时，优于无先验的RL和现有基于演示的方法。
原文摘要 · Abstract (English)
Large scale GPU-parallel reinforcement learning has changed what can be trained in robot simulation, yet most systems still optimize one specialist policy per task. We propose a construction methodology for turning structured manipulation task families into GPU-parallel multi-task RL benchmarks, and instantiate it as MT-Libero using LIBERO assets and task predicates in Isaac Lab. The resulting benchmark supports simultaneous reinforcement learning over heterogeneous task suites with parallel rendering, physics randomization, and state-input or visual-input policies. To make such training practical under sparse success signals and limited prior data, we further propose DGPO, an on-policy demonstration guided method that combines importance weighted PPO with adaptive behavior cloning on matched demonstration actions. DGPO enables a tunable preference toward demonstrated task distributions, outperforming both prior-free RL and existing demonstration-based methods while preserving the stability and online improvement benefits of on-policy PPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。