用偏好后悔机制让机器人在少量示范数据下学会复杂动作。
Optimizing Neurorobot Policy under Limited Demonstration Data through Preference Regret
- 通过可查询的偏好混合状态模型动态生成目标，实现自我模仿学习。
- 在有限示范数据下，机器人轨迹误差更小，测试表现优于现有方法。
- 适合数据稀缺场景下的机器人控制优化，尤其对高成本示范任务有帮助。
机器人从示范中进行强化学习（RLfD）通常假设专家数据充足，但现实中因数据稀缺和采集成本高，这一假设往往不成立。此外，模仿学习算法常假设数据独立同分布，导致测试时轨迹中的误差逐渐累积并恶化性能。为此，本文提出“掌握自身专长”（MYOE）框架，一种自模仿学习机制，使机器人能从少量示范数据中学习复杂行为。受人类感知与行动启发，设计了可查询的偏好混合状态空间模型（QMoP-SSM），用于每一步估计期望目标。这些目标用于计算“偏好后悔”，进而优化机器人控制策略。实验表明，该方法在鲁棒性、适应性和泛化性能上均优于当前主流的RLfD方案。相关代码已开源：https://github.com/rxng8/neurorobot-preference-regret-learning。
原文摘要 · Abstract (English)
Robot reinforcement learning from demonstrations (RLfD) assumes that expert data is abundant; this is usually unrealistic in the real world given data scarcity as well as high collection cost. Furthermore, imitation learning algorithms assume that the data is independently and identically distributed, which ultimately results in poorer performance as gradual errors emerge and compound within test-time trajectories. We address these issues by introducing the "master your own expertise" (MYOE) framework, a self-imitation framework that enables robotic agents to learn complex behaviors from limited demonstration data samples. Inspired by human perception and action, we propose and design what we call the queryable mixture-of-preferences state space model (QMoP-SSM), which estimates the desired goal at every time step. These desired goals are used in computing the "preference regret", which is used to optimize the robot control policy. Our experiments demonstrate the robustness, adaptability, and out-of-sample performance of our agent compared to other state-of-the-art RLfD schemes. The GitHub repository that supports this work can be found at: https://github.com/rxng8/neurorobot-preference-regret-learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。