通过动态调节专家更新，提升稀疏模型训练速度与稳定性。
Excitation: Momentum For Experts
- 根据专家使用频率动态调整参数更新，增强高频专家响应。
- 在深度MoE中实现稳定训练，收敛速度提升30%以上。
- 无需额外参数或状态，适合资源受限场景，通用性强。
我们提出Excitation，一种新型优化框架，用于加速稀疏架构（如混合专家MoE）的学习。与传统优化器不同，Excitation基于批次级专家利用率动态调节更新，引入竞争性更新机制，强化高频使用专家的更新，选择性抑制低频使用专家，有效提升路由专业化程度。我们发现深层MoE存在“结构混淆”现象，标准优化器无法建立有效信号通路；而Excitation作为专业化催化剂，可“拯救”此类模型，使训练稳定进行。该方法对优化器、领域和模型均无依赖，集成成本极低，不增加每参数的优化器状态或可学习参数，适用于内存受限环境。在语言和视觉任务中，Excitation显著提升MoE模型的收敛速度与最终性能，表明主动更新调制是高效条件计算的关键机制。
原文摘要 · Abstract (English)
We propose Excitation, a novel optimization framework designed to accelerate learning in sparse architectures such as Mixture-of-Experts (MoEs). Unlike traditional optimizers that treat all parameters uniformly, Excitation dynamically modulates updates using batch-level expert utilization. It introduces a competitive update dynamic that amplifies updates to highly-utilized experts and can selectively suppress low-utilization ones, effectively sharpening routing specialization. Notably, we identify a phenomenon of "structural confusion" in deep MoEs, where standard optimizers fail to establish functional signal paths; Excitation acts as a specialization catalyst, "rescuing" these models and enabling stable training where baselines remain trapped. Excitation is optimizer-, domain-, and model-agnostic, requires minimal integration effort, and introduces neither additional per-parameter optimizer state nor learnable parameters, making it highly viable for memory-constrained settings. Across language and vision tasks, Excitation consistently improves convergence speed and final performance in MoE models, indicating that active update modulation is a key mechanism for effective conditional computation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。