FoE架构让大模型推理更快，通信开销降低5倍以上。
Federation of Experts: Communication Efficient Distributed Inference for Large Language Models
- 将专家模块分组并行，减少跨设备通信
- 单机下无全量通信，多机时通信限于节点内
- 适合部署大规模语言模型的高效推理系统
混合专家（MoE）已成为提升大语言模型计算效率的主要机制。但在分布式环境下，专家间传递令牌嵌入成为显著瓶颈。本文提出新型联邦专家（FoE）架构，将Transformer层中的MoE块重构为多个专家集群，每个集群仅负责一个键值头，专家并行分布在各集群间。集群间通过求和同步注意力残差，驱动下一MoE块的路由与调度。在单机设置中，FoE完全消除了所有对所有通信，因同一组内的所有专家均位于同一GPU上。在多节点设置中，所有对所有通信被限制在节点内部总线，显著降低通信开销。在LongBench上的实现表明，FoE在单机与多机场景下均显著提升推理吞吐量与降低延迟：端到端前向传播延迟降低最多5.2倍，首次响应时间（TTFT）降低3.62倍，之后每字时间（TBT）降低1.95倍。同时，在相同规模与训练配置下，生成质量与标准MoE模型相当。
原文摘要 · Abstract (English)
Mixture of experts has emerged as the primary mechanism for making Large Language Models (LLMs) computationally efficient. However, in distributed settings, communicating token embeddings between experts is a significant bottleneck. We present the novel Federation of Experts (FoE) architecture. FoE restructures the MoE block of a transformer layer into multiple MoE clusters. Each cluster is responsible for only one of the KV heads and expert parallelism is applied between those experts. Between clusters, a sum synchronizes the post-attention residuals, which then drives routing and dispatch for the next MoE block. In a single-node setting, FoE completely eliminates all-to-all communication as all experts within a group are contained on the same GPU. In multi-node settings, FoE confines all-to-all communication to the intra-node fabric, thus significantly reducing communication overhead. An implementation of FoE finds that on LongBench, FoE significantly improves inference throughput and latency in both single-node and multi-node settings, reducing the end-to-end forward-pass latency by up to 5.2x, TTFT by 3.62x, and TBT by 1.95x. It does so while achieving comparable generation quality to a mixture of experts model of the same size and training configuration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。