让智能体高效完成可乱序执行的长序列任务
Reinforcement Learning for Long-Horizon Unordered Tasks: From Boolean to Coupled Reward Machines
- 用扩展版奖励机表达复杂任务，支持无序子任务
- 新算法QCoRM在多领域实验中显著提升长程任务效率
- 适合需要灵活调度的机器人、游戏等复杂任务场景
奖励机（RMs）能向强化学习智能体传达环境的奖励结构，支持非马尔可夫任务并提高采样效率。然而，对于子任务可任意顺序完成的长时程问题，传统方法面临信息量随子任务数量呈指数增长的挑战。本文提出三种RMs的泛化形式：(1) 数值型奖励机，以紧凑形式表达复杂任务；(2) 议程型奖励机，通过议程追踪待完成子任务；(3) 耦合型奖励机，为议程中的每个子任务配置耦合状态。此外，提出QCoRM算法——一种基于任务分解的Q-learning方法，利用耦合奖励机，并在表格设置下保持全局最优性保证。在四个不同领域（涵盖离散与连续动作及状态空间）的实验表明，当面对长时程无序子任务时，QCoRM的扩展性优于基线算法。
原文摘要 · Abstract (English)
Reward machines (RMs) inform reinforcement learning agents about the reward structure of the environment, enabling support for non-Markovian tasks and improving sample efficiency. However, learning with RMs is ill-suited for long-horizon problems where subtasks can be completed in any order. In such cases, the amount of information to learn increases exponentially with the number of unordered subtasks. We address this issue by introducing three generalisations of RMs: (1) Numeric RMs allow users to express complex tasks in a compact form. (2) In agenda RMs, states are associated with an agenda that tracks the remaining subtasks to complete. (3) Coupled RMs have coupled states associated with each subtask in the agenda. In addition, we introduce QCoRM, a new task-decomposition Q-learning-based algorithm that leverages coupled RMs and preserves global optimality guarantees in tabular settings. Our experiments across four domains -- featuring both discrete and continuous action and state spaces -- demonstrate that QCoRM scales better than baseline algorithms for long-horizon problems with unordered subtasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。