发现大模型深层专家更耐删，可大幅压缩而不降质。
Depth-Aware Sensitivity Analysis of Mixture-of-Experts Models via Magnitude-Based Expert Masking
- 按专家权重大小逐层屏蔽，系统测试各层重要性。
- 最后5层屏蔽640个专家仍保419/500高质量输出。
- 适合做模型压缩、轻量化部署与硬件优化的工程师。
Mixture-of-Experts (MoE) 架构通过稀疏激活实现大语言模型（LLMs）的高效扩展。本文针对 Qwen3.6-35B-A3B 模型（40 个 MoE 层，每层 256 个专家，Top-8 路由）在 XLCoST 跨语言代码翻译任务上开展分层敏感性分析，采用基于幅度的专家屏蔽策略。实验在三台 H100 GPU 服务器上覆盖 100、300 和 500 提示规模。结果表明：早期（0–9）和中期（10–29）层对专家屏蔽极为敏感，而晚期（30–39）尤其是极晚期（35–39）层可承受高强度屏蔽。全层屏蔽 30% 仅保留 150/300 高质量输出；而聚焦晚期的策略在屏蔽 640–1,145 个专家时仍保持 249–255/300 输出。在 500 提示验证集上，仅屏蔽最后 5 层（35–39）且掩码率 50% 的方案表现最佳，保留 419/500 高质量输出，仅屏蔽 640/10,240 个总专家。此外，将 Top-k 路由从 8 降至 6 无损失但未与剧烈屏蔽兼容。研究为深度感知的专家屏蔽提供实证基础，推动物理权重手术、激活评分与训练恢复技术发展。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) architectures scale large language models (LLMs) while preserving computational efficiency through sparse activation. Despite their widespread adoption, the relative importance of individual MoE layers remains insufficiently characterized, particularly for model compression. This paper presents a systematic layer-wise sensitivity analysis of the Qwen3.6-35B-A3B model (40 MoE layers, 256 experts per layer, top-8 routing) using magnitude-based expert masking on the XLCoST cross-lingual code translation benchmark. We conduct a multi-phase study spanning 100, 300, and 500 prompt evaluation scales across three H100 GPU servers. Our central finding is that layer sensitivity is strongly depth-dependent: early layers (0-9) and middle layers (10-29) are highly fragile to expert masking, while late layers (30-39), and especially very-late layers (35-39), tolerate aggressive masking of low-magnitude experts. Flat all-layer masking at 30% retains only 150/300 Good+Similar outputs at 300-prompt scale, whereas late-focused policies retain 249-255/300 while masking 640-1,145 experts. On a later 500-prompt held-out validation slice, the narrow very-late policy (layers 35-39 @ 50%) achieves the strongest quality/masked-expert tradeoff among tested candidates, retaining 419/500 Good+Similar outputs while masking only 640 of 10,240 total experts. We additionally characterize top-k routing width reduction from 8 to 6 active experts per token, which shows a large observed wall-clock reduction on a 100-prompt probe with no Good+Similar loss, though it does not yet compose cleanly with aggressive expert masking. These findings provide an empirical foundation for depth-aware MoE expert masking and establish a practical path toward physical weight surgery, activation-based expert scoring, and training-based recovery.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。