用大模型指导自动驾驶小模型,快速学会高效协作。
Language-Driven Policy Distillation for Cooperative Driving in Multi-Agent Reinforcement Learning
- 大模型生成决策示范,小模型通过梯度更新学习。
- 小模型在少量引导下快速提升,最终超越大模型性能。
- 适合需要高效协作的自动驾驶多智能体系统研究者。
车联网与自动驾驶车辆(CAVs)的协同驾驶技术对提升交通系统的效率和安全性至关重要。基于学习的方法,如多智能体强化学习(MARL),已在协同决策任务中展现出强大能力。然而,现有MARL方法仍存在学习效率低、性能不足的问题。近年来,大语言模型(LLMs)在各类序列决策任务中表现出显著能力。为提升协同智能体的学习能力,同时保证决策效率与成本效益,我们提出一种语言驱动的策略蒸馏方法(LDPD),用于引导MARL探索。该框架中,基于LLM的教师智能体通过自身决策示范训练更小的学生智能体,实现协同决策。教师智能体增强车联网的观测信息,并利用LLM进行复杂协同决策推理,结合精心设计的决策工具实现专家级决策,提供高质量教学经验。学生智能体则通过梯度策略更新将教师的先验知识提炼为自身模型。实验表明,学生智能体可在极少教师指导的情况下迅速提升能力,并最终超越教师性能。大量实验显示,本方法在性能和学习效率上均优于基线方法。
原文摘要 · Abstract (English)
The cooperative driving technology of Connected and Autonomous Vehicles (CAVs) is crucial for improving the efficiency and safety of transportation systems. Learning-based methods, such as Multi-Agent Reinforcement Learning (MARL), have demonstrated strong capabilities in cooperative decision-making tasks. However, existing MARL approaches still face challenges in terms of learning efficiency and performance. In recent years, Large Language Models (LLMs) have rapidly advanced and shown remarkable abilities in various sequential decision-making tasks. To enhance the learning capabilities of cooperative agents while ensuring decision-making efficiency and cost-effectiveness, we propose LDPD, a language-driven policy distillation method for guiding MARL exploration. In this framework, a teacher agent based on LLM trains smaller student agents to achieve cooperative decision-making through its own decision-making demonstrations. The teacher agent enhances the observation information of CAVs and utilizes LLMs to perform complex cooperative decision-making reasoning, which also leverages carefully designed decision-making tools to achieve expert-level decisions, providing high-quality teaching experiences. The student agent then refines the teacher's prior knowledge into its own model through gradient policy updates. The experiments demonstrate that the students can rapidly improve their capabilities with minimal guidance from the teacher and eventually surpass the teacher's performance. Extensive experiments show that our approach demonstrates better performance and learning efficiency compared to baseline methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。