用预训练规划器指导强化学习，提升部分可观测环境下的控制性能。
PriPG-RL: Privileged Planner-Guided Reinforcement Learning for Partially Observable Systems with Anytime-Feasible MPC
- 训练时让规划器提供全局信息，指导智能体学习
- 在仿真和真实四足机器人上实现高效导航
- 适合需要高鲁棒性的机器人控制场景
本文针对部分可观测系统中的强化学习训练问题，提出利用仅在训练阶段可用的特权、即时可行的规划器代理来提升性能。将该问题形式化为部分可观测马尔可夫决策过程（POMDP），其中拥有近似动力学模型和特权状态信息的规划器代理，指导仅观测到真实状态损失投影的学习代理。为此，我们设计了一种即时可行的模型预测控制（MPC）算法作为规划器代理。对于学习代理，提出规划器到策略软演员-评论家（P2P-SAC）方法，将规划器的特权知识蒸馏给学习器，以缓解部分可观测性，从而提升样本效率与最终策略性能。论文提供了严格的理论分析，并在 NVIDIA Isaac Lab 仿真环境中验证了该方法，成功部署于真实世界单位树 Go2 四足机器人，在复杂障碍环境中实现有效导航。
原文摘要 · Abstract (English)
This paper addresses the problem of training a reinforcement learning (RL) policy under partial observability by exploiting a privileged, anytime-feasible planner agent available exclusively during training. We formalize this as a Partially Observable Markov Decision Process (POMDP) in which a planner agent with access to an approximate dynamical model and privileged state information guides a learning agent that observes only a lossy projection of the true state. To realize this framework, we introduce an anytime-feasible Model Predictive Control (MPC) algorithm that serves as the planner agent. For the learning agent, we propose Planner-to-Policy Soft Actor-Critic (P2P-SAC), a method that distills the planner agent's privileged knowledge to mitigate partial observability and thereby improve both sample efficiency and final policy performance. We support this framework with rigorous theoretical analysis. Finally, we validate our approach in simulation using NVIDIA Isaac Lab and successfully deploy it on a real-world Unitree Go2 quadruped navigating complex, obstacle-rich environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。