用逻辑规则加速多智能体任务学习，显著缩短收敛时间。
Logic-based Task Representation and Reward Shaping in Multiagent Reinforcement Learning
- 将LTL任务转为自动机，构建动态SMDP进行无模型学习。
- 引入奖励塑造，使多智能体系统收敛时间大幅减少。
- 适合状态动作空间大的复杂多智能体场景。
本文提出一种基于线性时序逻辑（LTL）的任务表示与奖励塑造方法，以加速多智能体系统中最优策略的学习。给定每个智能体的选项（时序抽象动作），将任务规范转换为对应的布赫自动机，并采用无模型方法在线收集转移样本，动态构建产品半马尔可夫决策过程（SMDP）。在此基础上，使用基于价值的强化学习算法合成“正确设计”的控制器，无需学习系统的底层转移模型。针对多智能体带来的指数级样本复杂度问题，提出一种新型奖励塑造机制。在确定性网格世界仿真中测试不同任务，结果显示该方法显著降低收敛时间。同时发现，随着状态和动作空间增大，使用选项的优势愈发明显。
原文摘要 · Abstract (English)
This paper presents an approach for accelerated learning of optimal plans for a given task represented using Linear Temporal Logic (LTL) in multi-agent systems. Given a set of options (temporally abstract actions) available to each agent, we convert the task specification into the corresponding Buchi Automaton and proceed with a model-free approach which collects transition samples and constructs a product Semi Markov Decision Process (SMDP) on-the-fly. Value-based Reinforcement Learning algorithms can then be used to synthesize a correct-by-design controller without learning the underlying transition model of the multi-agent system. The exponential sample complexity due to multiple agents is dealt with using a novel reward shaping approach. We test the proposed algorithm in a deterministic gridworld simulation for different tasks and find that the reward shaping results in significant reduction in convergence times. We also infer that using options becomes increasing more relevant as the state and action space increases in multi-agent systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。