用因果图加速强化学习，让智能体更快理解复杂任务的时序奖励。
Expediting Reinforcement Learning by Incorporating Knowledge About Temporal Causality in the Environment
- 将时序逻辑因果图融入奖励形式化，捕捉环境中的时间依赖关系。
- 在多个任务上实现更快收敛，显著减少训练所需的交互次数。
- 适合需要快速迁移任务规范的智能体设计者，尤其适用于复杂时序任务。
强化学习算法在奖励稀疏且依赖复杂事件序列的任务中表现不佳。概率奖励机（PRMs）是能捕捉奖励信号中时序依赖和非确定性任务结果的有限状态形式化方法。尽管特定强化学习算法可利用其有限状态结构加速学习，但手动修改和设计PRMs仍很困难，阻碍了高层因果知识的利用以及奖励形式化向具有不同因果结构的新领域的迁移。本文提出一种新方法，将基于时序逻辑的因果图作为因果信息引入奖励形式化，从而加速策略学习并促进任务规范的跨环境迁移。此外，我们提供了该方法收敛至最优策略的理论结果，并通过实验验证了其有效性。
原文摘要 · Abstract (English)
Reinforcement learning (RL) algorithms struggle with learning optimal policies for tasks where reward feedback is sparse and depends on a complex sequence of events in the environment. Probabilistic reward machines (PRMs) are finite-state formalisms that can capture temporal dependencies in the reward signal, along with nondeterministic task outcomes. While special RL algorithms can exploit this finite-state structure to expedite learning, PRMs remain difficult to modify and design by hand. This hinders the already difficult tasks of utilizing high-level causal knowledge about the environment, and transferring the reward formalism into a new domain with a different causal structure. This paper proposes a novel method to incorporate causal information in the form of Temporal Logic-based Causal Diagrams into the reward formalism, thereby expediting policy learning and aiding the transfer of task specifications to new environments. Furthermore, we provide a theoretical result about convergence to optimal policy for our method, and demonstrate its strengths empirically.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。