arXiv:2606.15716cs.LG2026-06被引 2

提出统一框架,指导如何为不同目标选择专家剪枝准则。

How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle

论文配图:How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle
图 1 · 摘自论文原文
  • 构建路由频率、门控权重、激活强度的统一剪枝框架
  • 新准则MAN/MSAN在16个基准上平均提升8.8分,排名前二
  • 明确任务无关与任务相关剪枝应采用不同标准

混合专家(MoE)语言模型通过稀疏专家激活降低每标记计算量,但部署仍需存储全部专家池,因此一次性专家剪枝是减少内存占用的实用方法。尽管有效,现有剪枝准则多为启发式,无单一准则普遍最优。因此,建立适配不同部署目标的剪枝准则选择原则,仍是重要且研究不足的问题。本文围绕路由频率、门控权重和激活强度三个因素,提出统一的一次性MoE专家剪枝公式,得出选择原则:任务无关剪枝应优先使用基于路由令牌平均、无门控的激活准则;任务相关剪枝可保留路由频率与门控权重信息。该框架系统梳理了现有启发式准则,并提出两种新的任务无关准则:均值激活范数(MAN)与均值平方激活范数(MSAN)。在四个代表性MoE模型和16个多样化基准上,MAN与MSAN表现稳定,平均排名前二,相较最强基线平均性能提升最高达8.8分。

原文摘要 · Abstract (English)

Mixture-of-Experts (MoE) language models reduce per-token computation through sparse expert activation, yet deployment still requires storing the full expert pool, making one-shot expert pruning a practical approach for reducing memory usage. Although effective, existing criteria are largely heuristic, and no single criterion is universally optimal. Thus, establishing a principle for selecting pruning criteria suited to different deployment objectives remains an important yet largely underexplored problem in one-shot expert pruning. To this end, we introduce a unified formulation for one-shot MoE expert pruning organized around three factors: routing frequency, gate weighting, and activation strength. The formulation yields a criteria selection principle: task-agnostic pruning should favor routed-token-averaged, gate-free activation-based criteria, whereas task-specific pruning can benefit from retaining routing-frequency and gate-weight information. Beyond this principle, the formulation also provides a systematic view of existing heuristic criteria and gives rise to two new task-agnostic criteria, Mean Activation Norm (MAN) and Mean Squared Activation Norm (MSAN). Across four representative MoE models and 16 diverse benchmarks, MAN and MSAN are consistently strong in the task-agnostic setting, obtain the top-two average ranks, and improve average performance by up to 8.8 points over the strongest baseline.

专家模型剪枝MoE准则选择

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。