arXiv:2503.02279cs.LG2025-03被引 4

用梦境模型提升交通信号控制的效率,减少训练所需环境交互。

DreamerV3 for Traffic Signal Control: Hyperparameter Tuning and Performance

  • 基于世界模型的DreamerV3算法,通过想象预测优化决策。
  • 小模型配合中等训练比例可显著缩短调参时间。
  • 同一组参数适用于多种交通场景,适合实际部署。

强化学习(RL)已成为智能交通信号控制(TSC)策略研究的热门技术。然而,现有算法需大量与环境交互才能学习有效策略,难以应用于大规模任务。DreamerV3算法具备构建环境动态知识并基于过往经验预测未来结果的能力,通过想象训练减少真实交互。本文使用SUMO仿真平台,基于行程需求矩阵(OD matrix)在走廊式交通场景中训练基于DreamerV3的TSC模型,评估其性能。状态和奖励函数均基于排队长度设计,动作目标为有效管理队列。对训练比例和模型规模两个超参数进行系统调优与分析,发现较小模型搭配初始多个中等训练比例可大幅减少调参耗时;同时该方法在两种不同任务场景中仅用相同超参数即取得良好效果,具备广泛适用性。关于数据效率,尽管早期训练阶段奖励曲线波动剧烈,但大模型仍表现出轻微数据效率优势,且训练比例提升未显著加速收敛。

原文摘要 · Abstract (English)

Reinforcement learning (RL) has evolved into a widely investigated technology for the development of smart TSC strategies. However, current RL algorithms necessitate excessive interaction with the environment to learn effective policies, making them impractical for large-scale tasks. The DreamerV3 algorithm presents compelling properties for policy learning. It summarizes general dynamics knowledge about the environment and enables the prediction of future outcomes of potential actions from past experience, reducing the interaction with the environment through imagination training. In this paper, a corridor TSC model is trained using the DreamerV3 algorithm to explore the benefits of world models for TSC strategy learning. In RL environment design, to manage congestion levels effectively, both the state and reward functions are defined based on queue length, and the action is designed to manage queue length efficiently. Using the SUMO simulation platform, the two hyperparameters (training ratio and model size) of the DreamerV3 algorithm were tuned and analyzed across different OD matrix scenarios. We discovered that choosing a smaller model size and initially attempting several medium training ratios can significantly reduce the time spent on hyperparameter tuning. Additionally, we found that the approach is generally applicable as it can solve two TSC task scenarios with the same hyperparameters. Regarding the claimed data-efficiency of the DreamerV3 algorithm, due to the significant fluctuation of the episode reward curve in the early stages of training, it can only be confirmed that larger model sizes exhibit modest data-efficiency, and no evidence was found that increasing the training ratio accelerates convergence.

强化学习交通控制世界模型超参调优

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。