让稀疏专家模型获得密集反向传播，提升训练稳定性与性能
Dense Backpropagation Improves Training for Sparse Mixture-of-Experts
- 用历史输出均值作为缺失专家的默认输出，实现路由器的密集梯度更新
- 在多个场景下超越传统TopK路由，性能显著提升且计算开销极低
- 适合追求高效训练的稀疏模型研究者和工业部署团队
混合专家（MoE）预训练比稠密Transformer更具可扩展性，因其能将输入路由到少量激活的前馈参数。但这也导致仅接收稀疏反向传播，引发训练不稳和性能下降。本文提出轻量级近似方法Default MoE：将未激活专家的输出替换为训练过程中专家输出的指数移动平均值，使路由器对每个令牌都能接收到所有专家的信号。该方法在保持参数稀疏激活的同时，实现密集反向传播，显著提升训练表现。Default MoE在多种设置下优于标准TopK路由，且计算开销几乎不变。代码已开源：https://github.com/vatsal0/default-moe。
原文摘要 · Abstract (English)
Mixture of Experts (MoE) pretraining is more scalable than dense Transformer pretraining, because MoEs learn to route inputs to a sparse set of their feedforward parameters. However, this means that MoEs only receive a sparse backward update, leading to training instability and suboptimal performance. We present a lightweight approximation method that gives the MoE router a dense gradient update while continuing to sparsely activate its parameters. Our method, which we refer to as Default MoE, substitutes missing expert activations with default outputs consisting of an exponential moving average of expert outputs previously seen over the course of training. This allows the router to receive signals from every expert for each token, leading to significant improvements in training performance. Our Default MoE outperforms standard TopK routing in a variety of settings without requiring significant computational overhead. Code: https://github.com/vatsal0/default-moe.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。