无需校准，一键识别关键专家，显著降低模型部署开销。
AIMER: Calibration-Free Task-Agnostic MoE Expert Pruning
- 基于专家权重分布模式，提出无校准重要性评分方法
- 在7B至47B模型上实现更均衡的任务适应能力
- 仅需0.22-2.06秒即可完成全专家评分，适合实时部署
混合专家（MoE）语言模型在不增加每标记计算量的前提下提升参数容量，但部署时仍需存储完整专家池，因此专家剪枝对降低内存与服务开销至关重要。现有无任务依赖的专家剪枝方法通常依赖校准：通过校准集中的路由或激活统计估计专家重要性，导致剪枝决策受校准数据变化影响，并引入显著预处理成本。本文提出AIMER（绝对均值除以均方根重要性），一种简单且无需校准的判据，通过捕捉专家权重的集中模式识别更具区分度的专家，特别适用于无任务依赖的专家剪枝。在7B至47B不同架构的MoE语言模型及16个多样化基准上，AIMER始终优于现有无校准方法，在任务平衡性上表现更优；令人意外的是，其性能甚至超越基于广泛使用的无任务依赖C4语料校准的强基线方法，且仅需0.22–2.06秒即可完成所有专家评分。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) language models increase parameter capacity without proportional per-token computation, yet deployment still requires storing the full expert pool, making expert pruning important for reducing memory and serving overhead. Existing task-agnostic expert-pruning methods are typically calibration-dependent: they estimate expert importance from routing or activation statistics on a calibration set, making pruning decisions sensitive to calibration-data variation while introducing substantial preprocessing cost. We propose AIMER (\textbf{A}bsolute mean over root mean square \textbf{IM}portance for \textbf{E}xpert \textbf{R}anking), a simple calibration-free criterion that identifies more distinct experts by capturing the concentration pattern of expert weights, making it well suited for task-agnostic expert pruning. Across 7B to 47B MoE language models with distinct architectures and 16 diverse benchmarks, AIMER consistently delivers stronger capability balance across diverse tasks than existing calibration-free methods. Surprisingly, AIMER also achieves better balance than strong calibration-based expert-pruning baselines calibrated on the widely used task-agnostic C4 corpus, while requiring only 0.22--2.06 seconds to score all experts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。