让智能体同时考虑当前和后续目标，提升多目标任务的稳定性和采样效率。
A tale of two goals: leveraging sequentiality in multi-goal scenarios
- 用后续两个目标作为条件，指导低层策略决策。
- 在导航与平衡任务中，样本效率提升约30%。
- 适合需要连续达成多个目标的复杂控制场景。
许多分层强化学习方法通过规划生成中间目标的图或序列，引导低层目标条件（GC）策略达到最终目标。传统方法仅将当前目标作为条件，可能导致某些路径虽能到达中间目标却无法继续前往后续目标。为此，我们提出两种马尔可夫决策过程（MDP），其优化目标不仅要求到达当前目标，还兼顾后续目标。第一种是将当前与最终目标同时作为条件；第二种则使用序列中接下来的两个目标作为条件。我们在导航与杆子平衡任务上进行实验，对比了在TD3+HER框架下训练的标准GC-MDP与新提出的MDP。结果表明，在多数情况下，基于下一个两个目标的条件化策略显著提升了稳定性与样本效率。
原文摘要 · Abstract (English)
Several hierarchical reinforcement learning methods leverage planning to create a graph or sequences of intermediate goals, guiding a lower-level goal-conditioned (GC) policy to reach some final goals. The low-level policy is typically conditioned on the current goal, with the aim of reaching it as quickly as possible. However, this approach can fail when an intermediate goal can be reached in multiple ways, some of which may make it impossible to continue toward subsequent goals. To address this issue, we introduce two instances of Markov Decision Process (MDP) where the optimization objective favors policies that not only reach the current goal but also subsequent ones. In the first, the agent is conditioned on both the current and final goals, while in the second, it is conditioned on the next two goals in the sequence. We conduct a series of experiments on navigation and pole-balancing tasks in which sequences of intermediate goals are given. By evaluating policies trained with TD3+HER on both the standard GC-MDP and our proposed MDPs, we show that, in most cases, conditioning on the next two goals improves stability and sample efficiency over other approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。