不依赖训练和校准，动态跳过低贡献专家，提升MoE模型效率。
ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMs
- 通过全局谱代理与路由精调双机制，实时评估专家贡献度。
- 在50%跳过率下，困惑度降低7.96%，下游平均准确率提升4.15个百分点。
- 无需额外训练或校准，适合部署在资源受限的MoE大模型场景。
Mixture-of-Experts (MoE) 架构为扩展大语言模型提供了高效范式，但固定的 top-k 路由对每个 token 都激活相同数量的专家槽，造成大量冗余计算。现有专家跳过方法通常依赖路由器置信度、校准数据或额外训练，难以可靠估计被路由专家的实际贡献。为此，我们提出 ACE，一种无需训练、无需校准且保持检查点不变的 token 自适应专家跳过框架。ACE 包含两个互补组件：1）全局谱代理（GSP），从耦合的 gate、up 与 down 投影及 RMSNorm 缩放中估计全局变换能力；2）路由器条件精调（RCR），基于中心化路由器权重构建专家特定方向原型，并沿路由偏好方向评估专家响应。推理时，ACE 结合两者估计结果与运行时路由器门控，仅当二者均识别某专家为低贡献时才跳过，始终保留 top-1 专家。所有专家统计量离线计算，线上仅需查表与轻量级标量运算。在三个 MoE 基础大模型与八个基准上的实验表明，ACE 持续优于现有静态与动态基线，尤其在激进跳过率下优势更显著。例如，在 Qwen3.6-35B-A3B 上以 50% 跳过率运行时，ACE 将 WikiText-2 困惑度降低 7.96%，下游平均准确率较最强竞争方法提升 4.15 个百分点。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) architectures provide an efficient paradigm for scaling large language models (LLMs), yet fixed top-k routing activates the same number of expert slots for every token, causing substantial redundant computation. Existing expert-skipping methods often rely on router confidence, calibration data, or additional training, and therefore cannot reliably estimate the actual contribution of routed experts. To this end, we propose ACE, a training-free, calibration-free, and checkpoint-preserving framework for token-adaptive expert skipping in MoE-based LLMs. ACE contains two complementary components: 1) Global Spectral Proxy (GSP), which estimates global transformation capacity from the coupled gate, up, and down projections together with RMSNorm scaling; and 2) Router-Conditioned Refinement (RCR), which constructs expert-specific direction prototypes from centered router weights and evaluates expert responses along routing-preferred directions. During inference, ACE combines both estimates with runtime router gates and skips an expert slot only when both views identify it as low-contribution, while always retaining the top-1 expert. All expert statistics are computed offline, leaving only table lookups and lightweight scalar operations online. Extensive experiments across three MoE-based LLMs and eight benchmarks demonstrate that ACE consistently outperforms existing static and dynamic baselines, with increasingly pronounced advantages under aggressive expert skipping. For instance, at a 50% skipping ratio on Qwen3.6-35B-A3B, ACE reduces WikiText-2 perplexity by 7.96% and improves average downstream accuracy by 4.15 percentage points over the strongest competing method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。