用大模型生成无需训练的信用分配函数，提升MARL效果与可解释性
QLLM: Do We Really Need a Mixing Network for Credit Assignment in Multi-Agent Reinforcement Learning?
- 用大语言模型生成非线性信用分配函数，无需额外参数
- 在标准基准上性能优于基线，且参数更少
- 适用于多种值分解算法，适合追求高效可解释性的研究者
信用分配仍是多智能体强化学习（MARL）中的核心挑战，通常通过集中训练、分散执行（CTDE）范式下的值分解方法解决。然而现有值分解方法依赖预定义的混合网络，需额外训练，常导致信用分配不准确且可解释性差。本文提出QLLM框架，利用大语言模型（LLM）构建无需训练的信用分配函数（TFCAFs），其对全局状态呈非线性，提升可解释性且不引入额外可学习参数。采用编码器-评估器框架确保生成代码的正确性与可执行性。在标准MARL基准上的大量实验表明，QLLM持续优于基线，且所需可学习参数更少。此外，其在多种值分解算法间具有良好的泛化能力。代码已开源：https://github.com/MaoMaoLYJ/pymarl-qllm。
原文摘要 · Abstract (English)
Credit assignment remains a fundamental challenge in multi agent reinforcement learning (MARL) and is commonly addressed through value decomposition under the centralized training with decentralized ex ecution (CTDE) paradigm. However, existing value decomposition meth ods typically rely on predefined mixing networks that require additional training, often leading to imprecise credit attribution and limited in terpretability. We propose QLLM, a novel framework that leverages large language models (LLMs) to construct training-free credit assign ment functions (TFCAFs), where the TFCAFs are nonlinear with re spect to the global state and offer enhanced interpretability while intro ducing no extra learnable parameters. A coder-evaluator framework is employed to ensure the correctness and executability of the generated code. Extensive experiments on standard MARL benchmarks demon strate that QLLM consistently outperforms baselines while requiring fewer learnable parameters. Furthermore, it demonstrates generalization across a broad set of value decomposition algorithms. Code is available at https://github.com/MaoMaoLYJ/pymarl-qllm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。