让大模型在不同设备上实时切换轻重版本,节省内存还保持高精度。
FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models

- 按重要性排序专家通道,用动作学习实现可渐进裁剪的嵌套子模型。
- 不微调下保留99.8%性能,裁掉50%路由专家参数,40%预算微调后更优。
- 支持实时在线切换部署预算,硬件级优化提升吞吐与内存效率。
混合专家(MoE)语言模型通过稀疏激活专家扩展模型能力,已成为现代大模型的标准架构。然而,稀疏激活并未消除存储和部署所有专家的负担,且可用部署预算在不同设备、用户和负载间差异显著。现有MoE压缩方法多为固定预算,通常仅针对单一目标预算进行优化。本文研究一种新设定:将大型预训练MoE语言模型转化为跨预算的嵌套可部署子网络家族。方法首先按重要性对专家前馈通道排序,再让每个专家学习一个离散动作以裁剪自身通道。通过逐步增加成本压力,单次动作训练即可导出从高到低预算的一系列动作掩码,每个掩码对应一个可靠的小型子网络,且嵌套于排序后的基础模型中。此外,仅需在40%裁剪预算下进行一次恢复微调,即可恢复模型质量,并将优化后的模型迁移至其他未见预算。整体框架优于近期MoE压缩基线。具体而言,在Qwen2-57B-A14B上,无需微调即保留约99.8%基线性能,同时裁掉50%路由专家参数。部署时,裁剪后的子网络实现真实内存缩减与吞吐提升,并支持基于内核级协同设计的实时在线预算切换。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) language models scale model ability with sparsely activated experts, making this architecture a standard recipe for modern large models. However, sparse activation does not remove the deployment burden of storing and serving all experts, and the available deployment budget can vary substantially across devices, users, and workloads. Existing MoE compression methods are still largely fixed-budget, typically optimizing one compressed endpoint at each chosen target budget. We study a different setting: converting a large pretrained MoE LLM into a nested family of deployable subnetworks across budgets. Our method first ranks expert FFN channels by their importance, then lets each expert learn a discrete action to prune its channels. By gradually increasing cost pressure, a single action-training run exports a series of action masks from high to low budgets, each of which identifies a reliable smaller subnetwork nested in the ranked base model. Moreover, we use a single recovery fine-tune at a mid pruning budget (40%) to recover degraded model quality and transfer the recovered model to other unseen budgets. Overall, our framework surpasses recent MoE compression baselines. Specifically, on Qwen2-57B-A14B, our method retains ~99.8% of base performance while pruning 50% of routed expert parameters even without fine-tuning. For deployment, our pruned subnetworks deliver real memory reduction and throughput gains, and further support realtime online budget switching with kernel-level co-design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。