动态调整MoE模型各层专家激活数量，兼顾精度与计算效率。
Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEs
- 用元学习控制器按任务和层动态设定专家保留阈值和路由偏置。
- 平均仅需2.28个专家（减少62%）仍保持接近原精度，灵活权衡算力与性能。
- 训练好的控制器可跨任务迁移，无需重新训练，适合部署优化。
Mixture-of-Experts (MoE) 模型通过将每个标记路由到部分专家网络，在保持每标记计算稀疏的同时提升容量。现有方法中，活跃专家数量在各层和任务间固定，尽管层间角色与专家冗余随深度变化，任务难度也影响需求。现有方案仅解决部分问题：层级分配通常离线确定并用于所有任务；而标记级方法依赖局部路由信号，缺乏任务上下文。本文提出 MetaNet，一个支持集控制器，可为每层预测专家保留阈值和有界路由偏置。骨干网络、专家和路由器保持冻结。在 DeepSeek-MoE-16B-Chat 上，MetaNet 提供可调的准确率-专家激活权衡：相比固定 k=6，保守设置平均激活 3.61 个专家（减少40%），达到相近的 MMLU 准确率（0.489 vs. 0.474）；激进设置平均激活 2.28 个专家（减少62%），准确率约低 3.7 个百分点。该基于 MMLU 训练的控制器还可无重训练迁移到 C-Eval，平均激活 2.90 个专家（比固定 k=6 减少52%），准确率为 0.386。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models route each token to a subset of expert networks, increasing capacity while keeping per-token computation sparse. In many deployed MoEs, the number of active experts is fixed across layers and tasks, although layer roles and expert redundancy vary with depth and demand varies with difficulty. Existing approaches address only part of this setting: layer-wise allocations are usually determined offline and reused for all tasks, while token-level methods vary expert activation using local routing signals without task-level context. We propose MetaNet, a support-set controller that predicts, for each layer, an expert-retention threshold and a bounded routing bias. The backbone, experts, and router remain frozen. On DeepSeek-MoE-16B-Chat, MetaNet provides a tunable accuracy-expert-activation trade-off. Relative to fixed k=6, a conservative setting activates 3.61 experts on average (40% fewer) and achieves comparable MMLU accuracy (0.489 vs. 0.474), whereas an aggressive setting activates 2.28 experts on average (62% fewer) with accuracy approximately 3.7 percentage points lower. The MMLU-trained controller also transfers to C-Eval without retraining, activating 2.90 experts on average (52% fewer than fixed k=6) at 0.386 accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。