提出可保证目标达成的安全强化学习新方法,无需牺牲探索能力。
A novel agent with formal goal-reaching guarantees: an experimental study with a mobile robot
- 用批评者函数作为李雅普诺夫函数设计安全策略
- 在少样本实验中总成本低于PPO和改进SARSA
- 适合对安全性要求高的机器人控制场景
强化学习(RL)在机器人任务中表现高效便捷,但需探索大量状态-动作对,其中许多可能不安全或无关。在线无模型学习在缺乏目标状态可达性保障时可能危险且低效。现有安全方法常通过屏蔽系统限制动作至安全集,但易过度限制探索。本文提出新型安全无模型RL智能体CALF(Critic As Lyapunov Function),在保持所有状态-动作对可探索的同时,形式化保证目标状态的稳定到达。理论分析证明了其目标稳定性质,并在非完整轮式移动机器人TurtleBot3 Burger上进行了真实世界与数值实验。结果表明,在少样本设置下,CALF在总成本上优于成熟的PPO及改进版SARSA,验证了其高效性与安全性优势。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) has been shown to be effective and convenient for a number of tasks in robotics. However, it requires the exploration of a sufficiently large number of state-action pairs, many of which may be unsafe or unimportant. For instance, online model-free learning can be hazardous and inefficient in the absence of guarantees that a certain set of desired states will be reached during an episode. An increasingly common approach to address safety involves the addition of a shielding system that constrains the RL actions to a safe set of actions. In turn, a difficulty for such frameworks is how to effectively couple RL with the shielding system to make sure the exploration is not excessively restricted. This work presents a novel safe model-free RL agent called Critic As Lyapunov Function (CALF) and showcases how CALF can be used to improve upon control baselines in robotics in an efficient and convenient fashion while ensuring guarantees of stable goal reaching. The latter is a crucial part of safety, as seen generally. With CALF all state-action pairs remain explorable and yet reaching of desired goal states is formally guaranteed. Formal analysis is provided that shows the goal stabilization-ensuring properties of CALF and a set of real-world and numerical experiments with a non-holonomic wheeled mobile robot (WMR) TurtleBot3 Burger confirmed the superiority of CALF over such a well-established RL agent as proximal policy optimization (PPO), and a modified version of SARSA in a few-episode setting in terms of attained total cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。