从离线数据中学习分层策略，解决高层动作不可见难题
Offline Hierarchical Reinforcement Learning via Inverse Optimization
- 通过逆优化恢复隐藏的高层动作，重建可用数据集
- 在机器人和网络优化任务中显著优于端到端方法
- 适合有分层结构且数据静态的强化学习场景
分层策略在高维动作空间、长时序规划及稀疏奖励等序列决策问题中表现优异。然而，从静态离线数据中学习分层策略面临重大挑战：高层策略采取的动作在分层控制器中不可观测，且离线数据可能由不同策略结构生成，导致标准离线学习算法失效。本文提出OHIO框架，利用策略结构知识求解逆问题，恢复生成观测数据的潜在高层动作，构建适用于现成离线训练的数据集。我们在机器人控制与网络优化任务上验证该框架，结果表明其性能显著优于端到端强化学习方法，并提升鲁棒性。我们还考察了框架在直接部署与在线微调中的多种实现方式。代码与数据见https://ohio-offline-hierarchical-rl.github.io
原文摘要 · Abstract (English)
Hierarchical policies enable strong performance in many sequential decision-making problems, such as those with high-dimensional action spaces, those requiring long-horizon planning, and settings with sparse rewards. However, learning hierarchical policies from static offline datasets presents a significant challenge. Crucially, actions taken by higher-level policies may not be directly observable within hierarchical controllers, and the offline dataset might have been generated using a different policy structure, hindering the use of standard offline learning algorithms. In this work, we propose OHIO: a framework for offline reinforcement learning (RL) of hierarchical policies. Our framework leverages knowledge of the policy structure to solve the \textit{inverse problem}, recovering the unobservable high-level actions that likely generated the observed data under our hierarchical policy. This approach constructs a dataset suitable for off-the-shelf offline training. We demonstrate our framework on robotic and network optimization problems and show that it substantially outperforms end-to-end RL methods and improves robustness. We investigate a variety of instantiations of our framework, both in direct deployment of policies trained offline and when online fine-tuning is performed. Code and data are available at https://ohio-offline-hierarchical-rl.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。