用大模型生成训练路径,让多智能体系统更适应复杂现实环境。
cMALC-D: Contextual Multi-Agent LLM-Guided Curriculum Learning with Diversity-Based Context Blending
- 用大模型生成有意义的训练序列,替代不稳定的奖励信号。
- 通过融合不同环境特征,新场景使模型泛化能力提升37%以上。
- 适合研究多智能体强化学习与真实世界部署的学者参考。
许多多智能体强化学习(MARL)算法在固定仿真环境中训练,导致在真实世界复杂不确定条件下表现脆弱。上下文多智能体强化学习(cMARL)通过环境上下文变量参数化,训练出对所有环境配置均有效的上下文无关策略。现有cMARL方法尝试使用课程学习来训练和评估上下文无关策略,但常依赖不可靠的代理信号(如价值估计或广义优势估计),这些信号在多智能体场景中因智能体间动态和部分可观测性而噪声大、不稳定。为此,我们提出上下文多智能体大模型引导的课程学习与基于多样性的上下文混合框架(cMALC-D),利用大语言模型(LLM)生成语义上合理的课程并提供更鲁棒的评估信号。为防止模式崩溃并促进探索,引入一种新颖的基于多样性的上下文混合机制,通过组合先前上下文特征生成新的训练场景。在交通信号控制领域实验表明,cMALC-D相比现有课程学习基线,在泛化能力和样本效率上均有显著提升。代码已开源:https://github.com/DaRL-LibSignal/cMALC-D。
原文摘要 · Abstract (English)
Many multi-agent reinforcement learning (MARL) algorithms are trained in fixed simulation environments, making them brittle when deployed in real-world scenarios with more complex and uncertain conditions. Contextual MARL (cMARL) addresses this by parameterizing environments with context variables and training a context-agnostic policy that performs well across all environment configurations. Existing cMARL methods attempt to use curriculum learning to help train and evaluate context-agnostic policies, but they often rely on unreliable proxy signals, such as value estimates or generalized advantage estimates that are noisy and unstable in multi-agent settings due to inter-agent dynamics and partial observability. To address these issues, we propose Contextual Multi-Agent LLM-Guided Curriculum Learning with Diversity-Based Context Blending (cMALC-D), a framework that uses Large Language Models (LLMs) to generate semantically meaningful curricula and provide a more robust evaluation signal. To prevent mode collapse and encourage exploration, we introduce a novel diversity-based context blending mechanism that creates new training scenarios by combining features from prior contexts. Experiments in traffic signal control domains demonstrate that cMALC-D significantly improves both generalization and sample efficiency compared to existing curriculum learning baselines. We provide code at https://github.com/DaRL-LibSignal/cMALC-D.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。