用多层级反馈提升复杂任务中的强化学习效果
Reinforcement Learning from Multi-level and Episodic Human Feedback
- 采用每轮结束时的评分反馈，替代传统二元偏好判断
- 算法实现亚线性失误率，在模拟中表现优异
- 适合处理非马尔可夫奖励、需整体评估的任务
设计有效的奖励函数一直是强化学习中的难题，尤其在非结构化环境中应对复杂任务时。为解决这一问题，研究者提出多种利用人类输入来定义或优化奖励函数的学习范式。其中，基于人类反馈的强化学习通过人类对行为的偏好比较来构建奖励信号。与之不同，本文探索了多层级人类反馈——即在每轮结束后提供一个评分。这种反馈形式虽较粗粒度，但能提供比二元反馈更丰富的奖励函数信息，且适用于非马尔可夫奖励场景，因其基于整段轨迹的评估。本文提出一种高效算法，可同时学习奖励函数与最优策略，并证明该算法达到亚线性遗憾（sublinear regret），并通过大量模拟验证其有效性。
原文摘要 · Abstract (English)
Designing an effective reward function has long been a challenge in reinforcement learning, particularly for complex tasks in unstructured environments. To address this, various learning paradigms have emerged that leverage different forms of human input to specify or refine the reward function. Reinforcement learning from human feedback is a prominent approach that utilizes human comparative feedback, expressed as a preference for one behavior over another, to tackle this problem. In contrast to comparative feedback, we explore multi-level human feedback, which is provided in the form of a score at the end of each episode. This type of feedback offers more coarse but informative signals about the underlying reward function than binary feedback. Additionally, it can handle non-Markovian rewards, as it is based on the evaluation of an entire episode. We propose an algorithm to efficiently learn both the reward function and the optimal policy from this form of feedback. Moreover, we show that the proposed algorithm achieves sublinear regret and demonstrate its empirical effectiveness through extensive simulations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。