为大模型多智能体系统设计了新型强化微调方法,提升协作与推理能力。
MARFT: Multi-Agent Reinforcement Fine-Tuning
- 提出面向大模型多智能体的强化微调框架MARFT,适配异步交互与多样化架构。
- 在多个复杂任务上验证效果,显著优于传统多智能体强化学习方法。
- 开源实现支持研究复现,适合智能体协作、自主决策等方向的研究者。
基于大语言模型的多智能体系统(LaMAS)在需要多方面推理与协作的复杂任务中表现出强大能力,涵盖高质量演示生成到科学研究。尽管强化学习(RL)被广泛认为能提升智能体智能,但针对LaMAS的强化微调研究仍有限。直接应用传统多智能体强化学习(MARL)面临重大挑战,因其未能适配LaMAS的独特机制。本文系统研究了基于大模型的MARL,提出多智能体强化微调(MARFT)。引入与真实世界LaMAS优化对齐的灵活马尔可夫博弈(Flex-MG)形式化,并构建通用算法框架。分析从传统强化学习到强化微调(RFT)的演进,揭示经典MARL与MARFT在异步交互、个体特征感知设计及异构架构上的关键差异,推动面向LaMAS的RFT范式。提出稳健且可扩展的MARFT框架,详述其模块化算法,并提供开源实现以促进应用与后续研究。论文还讨论了动态环境建模、样本效率低和缺乏统一框架等开放挑战。通过连接理论基础与实践方法,本工作旨在为构建鲁棒、自适应、符合人类意图的智能体系统提供路线图。实现:https://github.com/jwliao-ai/MARFT。
原文摘要 · Abstract (English)
Large Language Model (LLM)-based Multi-Agent Systems (LaMAS) have demonstrated strong capabilities on complex agentic tasks requiring multifaceted reasoning and collaboration, from high-quality presentation generation to scientific research. Meanwhile, Reinforcement Learning (RL) is widely recognized for enhancing agent intelligence, but limited work has studied fine-tuning LaMAS with foundational RL techniques. Directly applying conventional Multi-Agent Reinforcement Learning (MARL) to LaMAS also introduces major challenges due to the unique mechanisms of LaMAS. To address these challenges, this article presents a comprehensive study of LLM-based MARL and proposes Multi-Agent Reinforcement Fine-Tuning (MARFT). We introduce Flex-MG, a new Markov Game formulation aligned with real-world LaMAS optimization, together with a universal algorithmic framework tailored to LaMAS. We review the evolution from traditional RL to Reinforcement Fine-Tuning (RFT), then analyze the multi-agent counterpart. For LaMAS, we identify key differences between classical MARL and MARFT, including asynchronous agent interactions, profile-aware agent design, and heterogeneous architectures. These differences motivate a LaMAS-oriented formulation of RFT. We present a robust and scalable MARFT framework, detail its modular algorithm, and provide an open-source implementation to support adoption and further research. The paper further discusses application perspectives and open challenges, including dynamic environment modeling, sample inefficiency, and the lack of cohesive frameworks. By connecting theoretical foundations with practical methodology, this work aims to serve as a roadmap for advancing MARFT toward resilient, adaptive, and human-aligned agentic systems. Implementation: https://github.com/jwliao-ai/MARFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。