发现专家内部稀疏性，无需修改模型即可大幅加速MoE推理。
Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution
- 利用专家内部激活稀疏性，跳过不活跃神经元计算。
- 在8个不同规模MoE模型中实现高达90%的内部稀疏度。
- 在vLLM上实现2.5倍层加速，适合高效部署大模型。
Mixture of Experts (MoE) 架构因其通过稀疏专家激活实现计算效率，已成为当前大语言模型的标准。然而，随着专家粒度细化，由于专家坍缩和负载不平衡等训练难题，进一步提升稀疏性变得愈发困难。本文探索并利用了MoE模型中尚未被充分挖掘的专家内部激活稀疏性这一补充维度。令人惊讶的是，在现有预训练的MoE模型中，无需修改激活函数或模型参数，即可直接获得显著的专家内部稀疏性，每个专家内可达90%的稀疏度，且精度损失可忽略。我们在8个现成的MoE模型（1B至400B参数）上验证了该现象，并扩展了vLLM的MoE执行流水线，通过跳过不活跃神经元的计算，在原有优化基础上实现了最高2.5倍的MoE层执行速度提升，以及相比原始密集vLLM基线1.2倍的端到端加速。
原文摘要 · Abstract (English)
Mixture of Experts (MoE) architecture has become the standard for state-of-the-art large language models, owing to its computational efficiency through sparse expert activation. However, sparsity through finer expert granularity is becoming increasingly difficult to achieve due to fundamental training challenges such as expert collapse and load imbalance. In this work, we explore and leverage intra-expert activation sparsity as a complementary and underexplored dimension of sparsity in MoE models. Surprisingly, substantial intra-expert sparsity is readily available in existing pre-trained MoE models, without any modification to the activation function or model parameters, providing up to 90% sparsity within each expert without significant accuracy loss. We explore intra-expert activation sparsity across eight off-the-shelf MoE models ranging from 1B to 400B parameters, and extend the MoE execution pipeline of vLLM to leverage intra-expert activation sparsity by skipping the computations of inactive neurons, on top of its existing optimizations, achieving up to 2.5 times speedup in MoE layer execution and 1.2 times end-to-end speedup compared to the original dense vLLM baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。