提出并行模块化框架ParMod，解决非马尔可夫任务的强化学习难题。
ParMod: A Parallel and Modular Framework for Learning Non-Markovian Tasks
- 将非马尔可夫任务按时序逻辑分解为多个子任务，由多个智能体并行训练。
- 在多个基准测试中性能优于现有方法，显著提升样本效率。
- 适合需要长期记忆和复杂时序约束的强化学习任务研究者。
常用的强化学习模型（马尔可夫决策过程）假设奖励仅依赖当前状态与动作，但许多现实任务具有非马尔可夫性，需长期记忆与依赖。此类任务面临奖励稀疏问题，学习难度更高。本文提出一种新颖的并行模块化强化学习框架ParMod，专用于由时序逻辑定义的非马尔可夫任务（NMT）。借助形式化技术，将NMT根据自动机结构（等价于其时序逻辑表示）分解为一系列子任务。在此基础上，多个智能体并行训练，每个负责一个子任务。除并行训练外，ParMod的核心在于：灵活的任务模块化方法，以及有效的奖励塑造机制以提升样本效率。在多个挑战性基准问题上进行综合评估，结果表明，ParMod在多种指标下均优于现有方法。本工作实现了强化学习、非马尔可夫任务与时序逻辑的良好结合。
原文摘要 · Abstract (English)
The commonly used Reinforcement Learning (RL) model, MDPs (Markov Decision Processes), has a basic premise that rewards depend on the current state and action only. However, many real-world tasks are non-Markovian, which has long-term memory and dependency. The reward sparseness problem is further amplified in non-Markovian scenarios. Hence learning a non-Markovian task (NMT) is inherently more difficult than learning a Markovian one. In this paper, we propose a novel \textbf{Par}allel and \textbf{Mod}ular RL framework, ParMod, specifically for learning NMTs specified by temporal logic. With the aid of formal techniques, the NMT is modulaized into a series of sub-tasks based on the automaton structure (equivalent to its temporal logic counterpart). On this basis, sub-tasks will be trained by a group of agents in a parallel fashion, with one agent handling one sub-task. Besides parallel training, the core of ParMod lies in: a flexible classification method for modularizing the NMT, and an effective reward shaping method for improving the sample efficiency. A comprehensive evaluation is conducted on several challenging benchmark problems with respect to various metrics. The experimental results show that ParMod achieves superior performance over other relevant studies. Our work thus provides a good synergy among RL, NMT and temporal logic.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。