arXiv:2411.13613cs.ROcs.AI2024-11ICRA

用李雅普诺夫指数设计机器人学习奖励,无需人工调参和随机初始化。

SuPLE: Robot Learning with Lyapunov Rewards

  • 基于系统动力学的李雅普诺夫指数生成内在奖励信号。
  • 在双摆杆等经典任务中实现无需辅助探索的稳定控制。
  • 适合真实机器人从自然起始状态学习,不依赖人为设定初始状态。

奖励函数是机器人学习中的关键组件,直接影响学习的样本与计算复杂度及解的质量。传统设计需领域知识,常不可得。本文利用系统动力学特性,无需额外假设,提出基于李雅普诺夫指数的内在奖励机制。具体而言,我们证明正李雅普诺夫指数之和(SuPLE)是理想的奖励候选。构建了该奖励的计算框架,并在多个典型动态系统样本化稳定任务中验证其有效性。该方法可消除对任意初始状态训练轨迹的依赖(即辅助探索),而此类做法在真实机器人中难以实现,因实际系统通常从自然静止状态(如倒垂摆、地面机器人)启动,无法随意重置。对比常见奖励函数,即使在双摆杆上翻并保持直立这一典型多连杆任务中,后者仍需辅助探索才能收敛;而SuPLE奖励无需此步骤即可成功。该方法为典型而非高度特化或微调场景下的有效机器人学习提供了新路径。代码已公开以支持复现与进一步研究。

原文摘要 · Abstract (English)

The reward function is an essential component in robot learning. Reward directly affects the sample and computational complexity of learning, and the quality of a solution. The design of informative rewards requires domain knowledge, which is not always available. We use the properties of the dynamics to produce system-appropriate reward without adding external assumptions. Specifically, we explore an approach to utilize the Lyapunov exponents of the system dynamics to generate a system-immanent reward. We demonstrate that the `Sum of the Positive Lyapunov Exponents' (SuPLE) is a strong candidate for the design of such a reward. We develop a computational framework for the derivation of this reward, and demonstrate its effectiveness on classical benchmarks for sample-based stabilization of various dynamical systems. It eliminates the need to start the training trajectories at arbitrary states, also known as auxiliary exploration. While the latter is a common practice in simulated robot learning, it is unpractical to consider to use it in real robotic systems, since they typically start from natural rest states such as a pendulum at the bottom, a robot on the ground, etc. and can not be easily initialized at arbitrary states. Comparing the performance of SuPLE to commonly-used reward functions, we observe that the latter fail to find a solution without auxiliary exploration, even for the task of swinging up the double pendulum and keeping it stable at the upright position, a prototypical scenario for multi-linked robots. SuPLE-induced rewards for robot learning offer a novel route for effective robot learning in typical as opposed to highly specialized or fine-tuned scenarios. Our code is publicly available for reproducibility and further research.

机器人学习奖励设计李雅普诺夫自适应控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。