通过专家激活轨迹优化剪枝，提升MoE模型效率与部署性。
MoE Pathfinder: Trajectory-driven Expert Pruning
- 将专家选择建模为全局路径规划问题，利用多信号融合评估
- 在多数任务上优于现有方法，实现非均匀层间专家保留
- 适合追求高效部署的LLM研究者与工程师
大型语言模型中的混合专家（MoE）架构在多样化任务中表现卓越，但面临部署复杂和激活效率低的问题。专家剪枝成为降低计算开销、简化部署的有前景方案。然而，现有方法通常依赖局部重要性度量，采用统一的层间剪枝策略，仅利用部分评估信号，忽视了不同层专家贡献的异质性。为此，我们提出基于专家激活轨迹的剪枝方法，将MoE视为加权计算图，将专家选择转化为全局最优路径规划问题。在此框架中，整合重构误差、路由概率和激活强度的互补重要性信号于轨迹层面，自然实现跨层非均匀专家保留。实验表明，该方法在几乎所有任务上均优于现有主流方法。
原文摘要 · Abstract (English)
Mixture-of-experts (MoE) architectures used in large language models (LLMs) achieve state-of-the-art performance across diverse tasks yet face practical challenges such as deployment complexity and low activation efficiency. Expert pruning has thus emerged as a promising solution to reduce computational overhead and simplify the deployment of MoE models. However, existing expert pruning approaches conventionally rely on local importance metrics and often apply uniform layer-wise pruning, leveraging only partial evaluation signals and overlooking the heterogeneous contributions of experts across layers. To address these limitations, we propose an expert pruning approach based on the trajectory of activated experts across layers, which treats MoE as a weighted computation graph and casts expert selection as a global optimal path planning problem. Within this framework, we integrate complementary importance signals from reconstruction error, routing probabilities, and activation strength at the trajectory level, which naturally yields non-uniform expert retention across layers. Experiments show that our approach achieves superior pruning performance on nearly all tasks compared with most existing approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。