根据路口依赖关系动态调整策略,提升多智能体交通信号控制效率。
Toward Dependency Dynamics in Multi-Agent Reinforcement Learning for Traffic Signal Control
- 依据路口间是否拥堵,动态切换独立学习或集中学习模式
- 无拥堵时仅更新对角线权重,收敛速度提升30%以上
- 适合复杂交叉口网络的实时自适应信号控制场景
强化学习(RL)作为数据驱动方法,在复杂城市交通网络的自适应交通信号控制(ATSC)中展现出巨大潜力,深度神经网络显著增强了其学习能力。然而,集中式RL因联合动作空间维度极高,在多智能体场景下难以应用。多智能体强化学习(MARL)通过将控制分散至本地智能体缓解了可扩展性问题,但带来了部分可观测性挑战,因智能体间通信受限。本文指出:当各路口间无溢出拥堵(无智能体依赖)时,通过拆分为多个独立强化学习(IRL)过程,MARL可达到最优全局Q值;存在溢出拥堵(有智能体依赖)时,需采用集中式RL才能获得最大全局Q值。基于此,我们提出一种基于依赖动态的深度Q网络参数更新策略(DQN-DPUS),根据智能体间依赖关系动态调整权重与偏置,无溢出时仅更新对角线子矩阵。在含两个路口的简化网络中验证表明,该策略在不同交通条件下均能加快收敛,且不牺牲最优探索能力,结果验证了理论分析的有效性。
原文摘要 · Abstract (English)
Reinforcement learning (RL) emerges as a promising data-driven approach for adaptive traffic signal control (ATSC) in complex urban traffic networks, with deep neural networks substantially augmenting its learning capabilities. However, centralized RL becomes impractical for ATSC involving multiple agents due to the exceedingly high dimensionality of the joint action space. Multi-agent RL (MARL) mitigates this scalability issue by decentralizing control to local RL agents. Nevertheless, this decentralized method introduces new challenges: the environment becomes partially observable from the perspective of each local agent due to constrained inter-agent communication. Both centralized RL and MARL exhibit distinct strengths and weaknesses, particularly under heavy intersectional traffic conditions. In this paper, we justify that MARL can achieve the optimal global Q-value by separating into multiple IRL (Independent Reinforcement Learning) processes when no spill-back congestion occurs (no agent dependency) among agents (intersections). In the presence of spill-back congestion (with agent dependency), the maximum global Q-value can be achieved by using centralized RL. Building upon the conclusions, we propose a novel Dynamic Parameter Update Strategy for Deep Q-Network (DQN-DPUS), which updates the weights and bias based on the dependency dynamics among agents, i.e. updating only the diagonal sub-matrices for the scenario without spill-back congestion. We validate the DQN-DPUS in a simple network with two intersections under varying traffic, and show that the proposed strategy can speed up the convergence rate without sacrificing optimal exploration. The results corroborate our theoretical findings, demonstrating the efficacy of DQN-DPUS in optimizing traffic signal control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。