用人类直觉提升强化学习采样效率,让机器人更智能且易理解。
SHIRE: Enhancing Sample Efficiency using Human Intuition in REinforcement Learning
- 用概率图模型编码人类对机器人任务的直觉,融入强化学习训练
- 在多个环境中实现25%-78%的采样效率提升,开销极低
- 提升策略可解释性,适合需要安全与透明性的机器人应用
神经网络在深度估计、光流计算、同步定位与建图(SLAM)及自动控制等机器人感知与控制任务中广泛应用。深度强化学习因其无需监督学习的高昂训练成本而被广泛采用,但其样本效率低下,需大量环境交互才能收敛。现有算法如Deep Q Learning和Soft Actor-Critic虽部分缓解此问题,却难以提供自主机器人应用所需的可解释性。人类对机器人中常见的长时序任务具有直观理解,合理利用该直觉可提升策略可解释性并增强样本效率。本文提出SHIRE框架,通过概率图模型(PGMs)编码人类直觉,并融入深度强化学习训练流程,显著提升采样效率。实验显示,该框架在多个评估环境中实现25%-78%的样本效率增益,且开销可忽略。此外,通过教授代理基础行为模式,增强了策略可解释性。真实世界演示进一步验证了所提方法的有效性。
原文摘要 · Abstract (English)
The ability of neural networks to perform robotic perception and control tasks such as depth and optical flow estimation, simultaneous localization and mapping (SLAM), and automatic control has led to their widespread adoption in recent years. Deep Reinforcement Learning has been used extensively in these settings, as it does not have the unsustainable training costs associated with supervised learning. However, DeepRL suffers from poor sample efficiency, i.e., it requires a large number of environmental interactions to converge to an acceptable solution. Modern RL algorithms such as Deep Q Learning and Soft Actor-Critic attempt to remedy this shortcoming but can not provide the explainability required in applications such as autonomous robotics. Humans intuitively understand the long-time-horizon sequential tasks common in robotics. Properly using such intuition can make RL policies more explainable while enhancing their sample efficiency. In this work, we propose SHIRE, a novel framework for encoding human intuition using Probabilistic Graphical Models (PGMs) and using it in the Deep RL training pipeline to enhance sample efficiency. Our framework achieves 25-78% sample efficiency gains across the environments we evaluate at negligible overhead cost. Additionally, by teaching RL agents the encoded elementary behavior, SHIRE enhances policy explainability. A real-world demonstration further highlights the efficacy of policies trained using our framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。