Lynx通过动态重映射降低MoE推理专家调用量，提升吞吐且不损失精度。
Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selection
- 利用专家亲和度分箱技术重分配每批低相关性任务，减少激活专家数。
- 在九个基准上实现最高1.30倍吞吐提升，精度损失低于1个百分点。
- 无需针对任务调优，可与现有优化技术协同使用，进一步提效。
由专家混合（MoE）模型提供的选择性参数激活使其成为现代基础模型的热门选择。然而，在服务场景中，批处理与MoE存在根本性矛盾：批处理迫使所有专家被激活，削弱了MoE的优势，并加剧内存带宽瓶颈。现有高效MoE推理方法即使经过大量工作负载定制化调优，仍无法解决此矛盾。本文提出LYNX系统，可在无需负载感知的前提下实现高效MoE推理。LYNX利用训练中负载均衡损失带来的批次级专家激活偏差与冗余，通过创新的亲和度分箱（AffinityBinning）技术，在每个批次内重新映射低亲和度的令牌-专家分配，从而减少总激活专家数量。在四个前沿模型家族、九个基准上的评估显示，LYNX在保持任务精度损失小于1%点的情况下，实现最高1.30倍吞吐提升。此外，LYNX与现有技术具有互补性，可使它们性能再提升最高1.38倍。
原文摘要 · Abstract (English)
Selective parameter activation provided by Mixture-of-Expert (MoE) models have made them a popular choice in modern foundational models. However, MoEs face a fundamental tension when employed for serving. Batching, critical for performance in serving, forces the activation of all experts, thereby negating MoEs' benefits and exacerbating memory bandwidth bottlenecks. Existing work on efficient MoE inference are unable to resolve this tension even with extensive workload-specific tuning. We present LYNX, a system that enables efficient MoE inference in a workload-agnostic fashion. LYNX leverages a key property of MoE training: load-balancing losses introduce batch-level expert activation skews and redundancy, which it exploits by remapping low-affinity token-to-expert assignments within each batch using a novel AffinityBinning technique that reduces the total experts invoked. Our evaluation of LYNX on four state-of-the-art model families across nine benchmarks shows that it achieves up to 1.30x improvement in throughput while maintaining accuracy loss of less than 1% points across tasks. Further, LYNX is complementary to existing techniques where it additionally boosts their performance by up to 1.38x.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。