让边缘设备上的大模型协作推理，提升响应质量且不依赖中心服务器。
Distributed Mixture-of-Agents for Edge Inference with Large Language Models
- 各设备独立运行模型，通过去中心化通信共享提示信息。
- 理论证明队列稳定条件，实验显示部分配置在AlpacaEval上更优。
- 适合资源受限的分布式智能终端场景，如手机、IoT设备。
Mixture-of-Agents (MoA) 近期被提出用于提升大型语言模型（LLMs）的性能，使多个独立的 LLM 协同推理，相比单一 LLM 能生成更优回复。本文研究在分布式环境下部署 MoA 架构，即 LLM 运行于各自关联用户的边缘设备上，每台设备具备独立计算能力。设备间通过去中心化的 gossip 算法通信，无需中心服务器。每个用户拥有专属的 LLM 模型，设备可交换自身用户提示或增强后的提示，以生成更精准的回答。当对应 LLM 忙时，用户提示暂存于设备队列中。鉴于边缘设备内存有限，需保证系统平均队列长度有界。本文在合理假设下，理论推导了设备队列的稳定性条件，并通过实验验证。进一步利用开源 LLM 实现分布式 MoA，实验表明某些配置在 AlpacaEval 2.0 上优于其他方案。代码已开源：https://github.com/purbeshmitra/distributed_moa。
原文摘要 · Abstract (English)
Mixture-of-Agents (MoA) has recently been proposed as a method to enhance performance of large language models (LLMs), enabling multiple individual LLMs to work together for collaborative inference. This collaborative approach results in improved responses to user prompts compared to relying on a single LLM. In this paper, we consider such an MoA architecture in a distributed setting, where LLMs operate on individual edge devices, each uniquely associated with a user and equipped with its own distributed computing power. These devices exchange information using decentralized gossip algorithms, allowing different device nodes to talk without the supervision of a centralized server. In the considered setup, different users have their own LLM models to address user prompts. Additionally, the devices gossip either their own user-specific prompts or augmented prompts to generate more refined answers to certain queries. User prompts are temporarily stored in the device queues when their corresponding LLMs are busy. Given the memory limitations of edge devices, it is crucial to ensure that the average queue sizes in the system remain bounded. In this paper, we address this by theoretically calculating the queuing stability conditions for the device queues under reasonable assumptions, which we validate experimentally as well. Further, we demonstrate through experiments, leveraging open-source LLMs for the implementation of distributed MoA, that certain MoA configurations produce higher-quality responses compared to others, as evaluated on AlpacaEval 2.0 benchmark. The implementation is available at: https://github.com/purbeshmitra/distributed_moa.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。