只在不确定时才要奖励，让强化学习更省成本。
Reducing Reward Dependence in RL Through Adaptive Confidence Discounting
- 根据模型置信度决定是否请求真实奖励，高信心时用预测替代。
- 实验显示仅需20%的奖励就能达到基线性能，节省大量资源。
- 适合人力或计算成本高的强化学习场景，如人机协作任务。
在人机协同强化学习或奖励计算成本高昂的环境中，频繁获取奖励会严重降低学习效率。由于获取人类反馈或计算昂贵奖励的代价高，每步都依赖真实奖励的算法在长训练过程中难以实施，限制了智能体的性能提升。本文提出一种新算法，仅当对当前状态中动作价值的认知不确定性较高时才请求真实奖励；高置信度时则使用奖励函数模型的预测值作为替代。通过减少对高成本奖励的依赖，该方法在奖励获取受限的场景下仍能高效学习。实验表明，本方法在返回值和学习所需回合数上与基线相当，但仅需基线20%的奖励次数即可达成目标。
原文摘要 · Abstract (English)
In human-in-the-loop reinforcement learning or environments where calculating a reward is expensive, the costly rewards can make learning efficiency challenging to achieve. The cost of obtaining feedback from humans or calculating expensive rewards means algorithms receiving feedback at every step of long training sessions may be infeasible, which may limit agents' abilities to efficiently improve performance. Our aim is to reduce the reliance of learning agents on humans or expensive rewards, improving the efficiency of learning while maintaining the quality of the learned policy. We offer a novel reinforcement learning algorithm that requests a reward only when its knowledge of the value of actions in an environment state is low. Our approach uses a reward function model as a proxy for human-delivered or expensive rewards when confidence is high, and asks for those explicit rewards only when there is low confidence in the model's predicted rewards and/or action selection. By reducing dependence on the expensive-to-obtain rewards, we are able to learn efficiently in settings where the logistics or expense of obtaining rewards may otherwise prohibit it. In our experiments our approach obtains comparable performance to a baseline in terms of return and number of episodes required to learn, but achieves that performance with as few as 20% of the rewards.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。