用稀疏机制提升多智能体大模型效率与多样性
SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agents
- 引入响应选择与早停机制，实现智能体间稀疏交互
- 在推理等基准上性能持平，计算成本降低显著
- 适合追求高效多智能体系统的研究人员
尽管多智能体系统已被证明能显著提升大型语言模型在各类任务中的表现，但智能体间的密集交互可能影响其效率与多样性。为此，我们受稀疏专家混合（SMoE）启发，提出稀疏智能体混合（SMoA）框架，以提升多智能体大模型的效率与多样性。与全连接结构不同，SMoA引入新型响应选择和早停机制，对个体大模型智能体之间的信息流进行稀疏化处理，在性能与效率间取得平衡。此外，借鉴SMoE中专家多样性原则以实现负载均衡，我们为每个大模型智能体分配不同的角色描述，促进多样化和发散式思考。在推理、对齐与公平性基准上的大量实验表明，SMoA在性能上可媲美传统混合智能体方法，但计算开销显著降低。进一步分析显示，SMoA更具稳定性，具备更强的可扩展性，并通过超参数优化展现出巨大潜力。
原文摘要 · Abstract (English)
While multi-agent systems have been shown to significantly enhance the performance of Large Language Models (LLMs) across various tasks and applications, the dense interaction between scaling agents potentially hampers their efficiency and diversity. To address these challenges, we draw inspiration from the sparse mixture-of-agents (SMoE) and propose a sparse mixture-of-agents (SMoA) framework to improve the efficiency and diversity of multi-agent LLMs. Unlike completely connected structures, SMoA introduces novel Response Selection and Early Stopping mechanisms to sparsify information flows among individual LLM agents, striking a balance between performance and efficiency. Additionally, inspired by the expert diversity principle in SMoE frameworks for workload balance between experts, we assign distinct role descriptions to each LLM agent, fostering diverse and divergent thinking. Extensive experiments on reasoning, alignment, and fairness benchmarks demonstrate that SMoA achieves performance comparable to traditional mixture-of-agents approaches but with significantly lower computational costs. Further analysis reveals that SMoA is more stable, has a greater capacity to scale, and offers considerable potential through hyper-parameter optimization. Code and data will be available at: https://github.com/David-Li0406/SMoA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。