提出MAESTRO方法，高效压缩MoE模型专家冗余，提升多任务泛化能力。
It Takes a MAESTRO To Prune Bad Experts
- 将专家激活路径建模为马尔可夫链，捕捉跨层依赖关系以评估重要性
- 在50%压缩率下平均性能保留率达93.2%，优于现有方法10.61%
- 适合需要轻量化部署且兼顾多领域表现的MoE模型应用
稀疏激活的专家混合（MoE）语言模型通过每标记仅激活少量参数实现显著推理效率提升，但所有专家始终驻留内存，构成部署瓶颈。现有结构化剪枝方法多针对密集型Transformer设计，依赖局部启发式判断专家重要性，忽视了MoE路由的相互依赖特性。我们提出MAESTRO（基于转移路由的马尔可夫链近似专家稀疏化），将自回归专家激活轨迹建模为遍历马尔可夫链，其平稳分布编码跨层依赖关系，生成全局感知的重要性度量。在包含安全、偏见与伦理在内的五个不同领域上评估，MAESTRO在严格50%压缩率下平均性能保留率达93.2%，相较最优基线提升最高达10.61%，且跨任务方差显著降低，表明全局一致、路由对齐的剪枝能提升模型在异构任务间的稳定性与泛化能力。
原文摘要 · Abstract (English)
Sparsely-activated Mixture-of-Experts (MoE) language models achieve remarkable inference efficiency by activating only a small fraction of parameters per token, yet their full expert banks reside in memory at all times, creating a prohibitive deployment bottleneck. Existing structured pruning methods, largely designed for dense transformers, assess expert importance using locally derived heuristics that are blind to the interdependent nature of MoE routing. We introduce MAESTRO (Markov-chain Approximated Expert Sparsification via Transition-based ROuting), a structured pruning framework designed for MoE architectures that models autoregressive expert activation trajectories as Ergodic Markov chains whose stationary distributions encode cross-layer dependencies, yielding a globally aware importance heuristic. Evaluated across five diverse domains including Safety, Bias, and Ethics, MAESTRO outperforms state-of-the-art baselines by up to 10.61% in average performance retention under a strict 50% compression regime, while exhibiting substantially lower cross-task variance, indicating that global, routing-congruent pruning produces models that generalize more consistently across heterogeneous tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。