提出两种去中心化大模型协作方法，提升复杂任务下的训练效率与稳定性。
Learning Decentralized LLM Collaboration with Multi-Agent Actor Critic
- 设计去中心化与集中式评论者两种协同训练框架，支持并行推理部署。
- 在长周期或稀疏奖励任务中，去中心化方法收敛更差，需更多样本。
- 适合需要灵活部署的多智能体协作场景，如代码生成、游戏策略制定。
近期研究探索了通过多智能体强化学习（MARL）优化大模型协作。然而，多数MARL微调方法依赖预设执行协议，通常需要集中式执行。去中心化协作在实践中更具吸引力，因智能体可并行推理且部署灵活。此外，现有方法采用蒙特卡洛法进行微调，存在方差高问题，需更多样本才能有效训练。而演员-评论者方法在MARL中广泛应用以缓解此问题，因此我们开发了多智能体演员-评论者（MAAC）方法以优化去中心化大模型协作。本文分析了这些MAAC方法适用的时机与原因。提出两种方法：CoLLM-CC（集中式评论者）与CoLLM-DC（去中心式评论者）。在写作、编程和游戏领域实验表明，在短周期和密集奖励设置下，蒙特卡洛方法与CoLLM-DC性能可媲美CoLLM-CC；但在长周期或稀疏奖励任务中，两者均表现逊于CoLLM-CC，其中蒙特卡洛方法需显著更多样本，而CoLLM-DC难以收敛。
原文摘要 · Abstract (English)
Recent work has explored optimizing LLM collaboration through Multi-Agent Reinforcement Learning (MARL). However, most MARL fine-tuning approaches rely on predefined execution protocols, which often require centralized execution. Decentralized LLM collaboration is more appealing in practice, as agents can run inference in parallel with flexible deployments. Also, current approaches use Monte Carlo methods for fine-tuning, which suffer from high variance and thus require more samples to train effectively. Actor-critic methods are prevalent in MARL for dealing with these issues; thus, we developed Multi-Agent Actor-Critic (MAAC) methods to optimize decentralized LLM collaboration. In this paper, we analyze when and why these MAAC methods are beneficial. We propose 2 MAAC approaches, \textbf{CoLLM-CC} with a \textbf{C}entralized \textbf{C}ritic and \textbf{CoLLM-DC} with \textbf{D}ecentralized \textbf{C}ritics. Our experiments across writing, coding, and game-playing domains show that Monte Carlo methods and CoLLM-DC can achieve performance comparable to CoLLM-CC in short-horizon and dense-reward settings. However, they both underperform CoLLM-CC on long-horizon or sparse-reward tasks, where Monte Carlo methods require substantially more samples and CoLLM-DC struggles to converge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。