提出可扩展的分层强化学习方法，显著提升高吞吐环境下的训练效率。
Scalable Option Learning in High-Throughput Environments
- 设计SOL算法，通过优化架构实现分层强化学习的高吞吐训练。
- 在NetHack上使用300亿帧数据训练，吞吐量提升约35倍。
- 适用于复杂游戏和物理模拟环境，适合大规模RL研究者使用。
分层强化学习（Hierarchical RL）具备实现长时程有效决策的潜力。现有方法虽具前景，却尚未实现大规模训练的优势。本文识别并解决了将在线分层RL扩展至高吞吐环境中的若干关键挑战。提出可扩展选项学习（Scalable Option Learning, SOL），一种高度可扩展的分层强化学习算法，在相同条件下相比现有分层方法实现约35倍的吞吐量提升。为验证SOL的性能与可扩展性，我们在复杂游戏NetHack上使用300亿帧经验进行训练，显著超越扁平化智能体，并展现出正向扩展趋势。同时在MiniHack和Mujoco环境中验证其通用性。代码已开源：github.com/facebookresearch/sol。
原文摘要 · Abstract (English)
Hierarchical reinforcement learning (RL) has the potential to enable effective decision-making over long timescales. Existing approaches, while promising, have yet to realize the benefits of large-scale training. In this work, we identify and solve several key challenges in scaling online hierarchical RL to high-throughput environments. We propose Scalable Option Learning (SOL), a highly scalable hierarchical RL algorithm which achieves a ~35x higher throughput compared to existing hierarchical methods. To demonstrate SOL's performance and scalability, we train hierarchical agents using 30 billion frames of experience on the complex game of NetHack, significantly surpassing flat agents and demonstrating positive scaling trends. We also validate SOL on MiniHack and Mujoco environments, showcasing its general applicability. Our code is open sourced at: github.com/facebookresearch/sol.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。