用动态奖励设计让强化学习更好完成复杂任务
Adaptive Reward Design for Reinforcement Learning
- 基于LTL公式设计可自适应调整的奖励函数
- 在多个基准环境上提前收敛，任务完成率更高
- 适合需要分步达成目标的复杂任务场景
近年来，使用线性时序逻辑（LTL）精确描述复杂任务并生成强化学习（RL）奖励函数受到广泛关注。然而，现有方法多采用稀疏奖励（仅在任务完成时给予1分，否则为0），无法激励中间子任务的完成。尤其在存在不确定性环境中，即使进展顺利，任务完成仍不可靠。为此，我们提出一套奖励函数，鼓励代理尽可能完成由LTL公式指定的任务，并开发了一种在学习过程中动态更新奖励的自适应奖励塑造方法。在多个基准强化学习环境中的实验表明，该方法普遍优于基线，能更早收敛至表现更优的策略，获得更高的期望回报和任务完成率。
原文摘要 · Abstract (English)
There is a surge of interest in using formal languages such as Linear Temporal Logic (LTL) to precisely and succinctly specify complex tasks and derive reward functions for Reinforcement Learning (RL). However, existing methods often assign sparse rewards (e.g., giving a reward of 1 only if a task is completed and 0 otherwise). By providing feedback solely upon task completion, these methods fail to encourage successful subtask completion. This is particularly problematic in environments with inherent uncertainty, where task completion may be unreliable despite progress on intermediate goals. To address this limitation, we propose a suite of reward functions that incentivize an RL agent to complete a task specified by an LTL formula as much as possible, and develop an adaptive reward shaping approach that dynamically updates reward functions during the learning process. Experimental results on a range of benchmark RL environments demonstrate that the proposed approach generally outperforms baselines, achieving earlier convergence to a better policy with higher expected return and task completion rate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。