用大模型自动设计车队协同的奖励函数,提升强化学习效果。
An Automated Reinforcement Learning Reward Design Framework with Large Language Model for Cooperative Platoon Coordination
- 用大模型分析代码和任务,生成初始奖励函数。
- 通过迭代优化,使强化学习性能平均提升10%。
- 适合需要快速部署智能交通系统的研究人员。
强化学习在车队协同问题中展现出卓越决策能力,但因协同目标多变、决策复杂且人工设计奖励函数耗时,难以获得高效奖励函数。本文正式定义了车队协同奖励设计问题(PCRDP),提出基于大语言模型的自动奖励设计框架PCRD,通过LLM驱动的初始化与迭代优化,实现奖励函数的系统化生成。该框架包含分析与初始奖励(AIR)模块,引导大模型通过思维链理解环境代码与任务要求,降低生成幻觉风险;以及进化模块,根据训练反馈对奖励函数进行微调与重构,平衡探索多样性与收敛稳定性。为验证方法,在长江三角洲交通网络仿真中构建六种不同复杂度的挑战性协同场景。对比实验表明,使用PCRD生成奖励函数的强化学习智能体在所有场景中均优于人工设计,平均性能指标提升10%。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) has demonstrated excellent decision-making potential in platoon coordination problems. However, due to the variability of coordination goals, the complexity of the decision problem, and the time-consumption of trial-and-error in manual design, finding a well performance reward function to guide RL training to solve complex platoon coordination problems remains challenging. In this paper, we formally define the Platoon Coordination Reward Design Problem (PCRDP), extending the RL-based cooperative platoon coordination problem to incorporate automated reward function generation. To address PCRDP, we propose a Large Language Model (LLM)-based Platoon coordination Reward Design (PCRD) framework, which systematically automates reward function discovery through LLM-driven initialization and iterative optimization. In this method, LLM first initializes reward functions based on environment code and task requirements with an Analysis and Initial Reward (AIR) module, and then iteratively optimizes them based on training feedback with an evolutionary module. The AIR module guides LLM to deepen their understanding of code and tasks through a chain of thought, effectively mitigating hallucination risks in code generation. The evolutionary module fine-tunes and reconstructs the reward function, achieving a balance between exploration diversity and convergence stability for training. To validate our approach, we establish six challenging coordination scenarios with varying complexity levels within the Yangtze River Delta transportation network simulation. Comparative experimental results demonstrate that RL agents utilizing PCRD-generated reward functions consistently outperform human-engineered reward functions, achieving an average of 10\% higher performance metrics in all scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。