构建机器人多任务强化学习大规模并行基准，加速算法评估与对比。
Benchmarking Massively Parallelized Multi-Task Reinforcement Learning for Robotics Tasks
- 设计基于IsaacGym的50个抓取+20个行走任务并行训练框架。
- 在1000个环境并行下实现比传统方法快4倍的训练速度。
- 首次系统评估大规模并行对多任务强化学习的影响，适合算法研究者使用。
多任务强化学习（MTRL）已成为应对复杂现实机器人任务的关键训练范式，需具备泛化性和鲁棒性的策略。与此同时，大规模并行训练因借助GPU加速仿真显著提升数据采集效率而日益流行，可通过并行模拟异构场景实现多任务多样化数据收集。然而，现有MTRL研究大多局限于低并行度下的离策略方法（如SAC）。MTRL可借助高渐近性能的同策略算法优势，其批次数据依赖当前策略，从而充分受益于GPU加速仿真提供的大规模并行能力。为弥合这一差距，我们提出大规模并行多任务机器人基准MTBench，包含50个操作任务和20个移动任务，基于GPU加速仿真器IsaacGym构建。该基准集成4种基础RL算法与7种先进MTRL算法架构，提供统一评估框架。大量实验表明，使用MTBench可大幅提升MTRL方法的评估效率，同时揭示了大规模并行与MTRL结合带来的独特挑战。代码开源地址：https://github.com/Viraj-Joshi/MTBench。
原文摘要 · Abstract (English)
Multi-task Reinforcement Learning (MTRL) has emerged as a critical training paradigm for applying reinforcement learning (RL) to a set of complex real-world robotic tasks, which demands a generalizable and robust policy. At the same time, \emph{massively parallelized training} has gained popularity, not only for significantly accelerating data collection through GPU-accelerated simulation but also for enabling diverse data collection across multiple tasks by simulating heterogeneous scenes in parallel. However, existing MTRL research has largely been limited to off-policy methods like SAC in the low-parallelization regime. MTRL could capitalize on the higher asymptotic performance of on-policy algorithms, whose batches require data from the current policy, and as a result, take advantage of massive parallelization offered by GPU-accelerated simulation. To bridge this gap, we introduce a massively parallelized $\textbf{M}$ulti-$\textbf{T}$ask $\textbf{Bench}$mark for robotics (MTBench), an open-sourced benchmark featuring a broad distribution of 50 manipulation tasks and 20 locomotion tasks, implemented using the GPU-accelerated simulator IsaacGym. MTBench also includes four base RL algorithms combined with seven state-of-the-art MTRL algorithms and architectures, providing a unified framework for evaluating their performance. Our extensive experiments highlight the superior speed of evaluating MTRL approaches using MTBench, while also uncovering unique challenges that arise from combining massive parallelism with MTRL. Code is available at https://github.com/Viraj-Joshi/MTBench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。