让强化学习在40%反馈错误时仍能有效训练。
CANDERE-COACH: Reinforcement Learning from Noisy Feedback
- 通过噪声过滤机制处理教师反馈中的错误信号。
- 实验表明，即使40%反馈错误，模型仍可成功学习。
- 适合需要人类反馈但难以保证准确性的场景。
近年来，强化学习被广泛应用于诸多挑战性任务中。然而，其表现优异通常依赖于良好设计的奖励函数，而这类函数往往稀疏或需人工精心构建，易出错。引入人类先验知识被视为解决该问题的可行途径，如模仿学习、基于偏好的学习和逆强化学习。学习反馈是另一框架，使强化学习智能体可通过二元评价信号（正向或负向）学习教师对动作的评估。然而，这些方法常假设教师反馈完全准确，这一假设过于严格。现实中，反馈可能因教师经验有限、认知负荷、注意力不集中等因素而存在噪声。本文提出CANDERE-COACH算法，能够从非最优教师的噪声反馈中学习。我们设计了一种在线噪声过滤机制，有效净化反馈数据，使强化学习智能体在高达40%反馈错误的情况下仍能成功训练。在三个常见任务域上的实验验证了该方法的有效性。
原文摘要 · Abstract (English)
In recent times, Reinforcement learning (RL) has been widely applied to many challenging tasks. However, in order to perform well, it requires access to a good reward function which is often sparse or manually engineered with scope for error. Introducing human prior knowledge is often seen as a possible solution to the above-mentioned problem, such as imitation learning, learning from preference, and inverse reinforcement learning. Learning from feedback is another framework that enables an RL agent to learn from binary evaluative signals describing the teacher's (positive or negative) evaluation of the agent's action. However, these methods often make the assumption that evaluative teacher feedback is perfect, which is a restrictive assumption. In practice, such feedback can be noisy due to limited teacher expertise or other exacerbating factors like cognitive load, availability, distraction, etc. In this work, we propose the CANDERE-COACH algorithm, which is capable of learning from noisy feedback by a nonoptimal teacher. We propose a noise-filtering mechanism to de-noise online feedback data, thereby enabling the RL agent to successfully learn with up to 40% of the teacher feedback being incorrect. Experiments on three common domains demonstrate the effectiveness of the proposed approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。