用注意力机制生成密集奖励，让延迟稀疏奖励任务可训练
Attention-Based Reward Shaping for Sparse and Delayed Rewards
- 用Transformer注意力从历史轨迹中挖掘潜在奖励信号
- 在全延迟奖励环境下使强化学习成功训练，数据量可低至随机策略轨迹
- 完全离线、通用性强，适合各类复杂稀疏奖励场景
稀疏且延迟的奖励函数是现实强化学习应用的重大障碍。本文提出基于注意力的奖励塑造（ARES），一种通用且鲁棒的算法，利用Transformer的注意力机制生成塑造奖励，为任意环境构建密集奖励函数。ARES仅需一组轨迹及其最终回报作为输入，可完全离线训练，即使使用小规模数据集或随机策略产生的轨迹也能生成有意义的塑造奖励。ARES兼容任意强化学习算法，可处理任意程度的奖励稀疏性。实验聚焦最困难的情况：奖励仅在每轮结束时才给出。我们在多种环境、广泛使用的强化学习算法和基线方法上评估ARES，结果表明其能显著提升延迟奖励场景下的学习效果，使原本需要不切实际数据量甚至无法学习的任务变为可行。据我们所知，ARES是首个完全离线、对极端奖励延迟和低质量数据保持鲁棒、且不限于目标导向任务的方法。
原文摘要 · Abstract (English)
Sparse and delayed reward functions pose a significant obstacle for real-world Reinforcement Learning (RL) applications. In this work, we propose Attention-based REward Shaping (ARES), a general and robust algorithm which uses a transformer's attention mechanism to generate shaped rewards and create a dense reward function for any environment. ARES requires a set of episodes and their final returns as input. It can be trained entirely offline and is able to generate meaningful shaped rewards even when using small datasets or episodes produced by agents taking random actions. ARES is compatible with any RL algorithm and can handle any level of reward sparsity. In our experiments, we focus on the most challenging case where rewards are fully delayed until the end of each episode. We evaluate ARES across a diverse range of environments, widely used RL algorithms, and baseline methods to assess the effectiveness of the shaped rewards it produces. Our results show that ARES can significantly improve learning in delayed reward settings, enabling RL agents to train in scenarios that would otherwise require impractical amounts of data or even be unlearnable. To our knowledge, ARES is the first approach that works fully offline, remains robust to extreme reward delays and low-quality data, and is not limited to goal-based tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。