通过微调让专家模型更少激活，提升推理效率
MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models
- 微调专家模型使其更倾向于少激活专家
- 吞吐量提升1.2-3倍，最高达14.7倍
- 适合资源受限场景的高效专家模型推理
混合专家（MoE）模型架构可显著减少每令牌激活的参数量，实现高效的训练与推理。然而，其庞大的总体参数量和模型尺寸限制了在资源受限环境中的广泛应用，因为所有参数仍需加载到GPU内存中。此前工作通过将部分专家卸载至CPU内存，并在激活时才传回GPU来缓解内存瓶颈。但实际中，专家传输带来显著I/O延迟。我们提出MELINOE，一种对MoE模型进行微调的方法，使其更倾向于在每个序列中激活更少的专家。将这些偏好专家缓存于GPU内存中，可降低专家切换频率和跨设备传输开销。MELINOE相较于高效基线提升1.2-3倍吞吐量，相比高传输基线最高提升14.7倍，同时保持或甚至提升下游任务性能，是一种可靠的提升MoE推理效率的方法。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) model architectures can significantly reduce the number of activated parameters per token, enabling computationally efficient training and inference. However, their large overall parameter counts and model sizes have precluded their widespread usage in resource-constrained settings as all of the parameters must still be loaded into GPU memory. Prior works aim to address this memory bottleneck by offloading certain experts into CPU memory and porting them to GPU memory only when they are activated. In practice, these methods suffer from the significant I/O latency incurred by expert transfer. We present MELINOE, a method that fine-tunes an MoE model to more strongly prefer activating a smaller number of experts per sequence. Caching these preferred experts in GPU memory reduces expert churn and CPU-GPU transfer overhead. MELINOE increases throughput by $1.2-3\times$ over efficient baselines and up to $14.7\times$ over transfer-heavy baselines while retaining or even improving the performance of the model on a downstream task, making it a reliable method for improving MoE inference efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。