arXiv:2512.03915math.OCcs.AI2025-12被引 1

提出无辅助损失的专家负载均衡理论框架,解释为何能高效分配计算资源。

A Theoretical Framework for Auxiliary-Loss-Free Load Balancing of Sparse Mixture-of-Experts in Large-Scale AI Models

  • 将负载均衡视为单次常数时间求解分配问题的原始-对偶方法
  • 证明了目标函数单调提升与近似均衡的保证,且在线设置下具对数期望遗憾界
  • 适用于大规模AI模型训练,尤其适合追求高效部署的工程团队

在大规模AI训练中,稀疏混合专家(s-MoE)层通过仅激活每令牌少量专家实现可扩展性。其操作挑战在于负载均衡:如何路由令牌以最小化闲置专家数量,这对高效利用昂贵GPU及充分训练所有专家参数至关重要。本文为DeepSeek的Wang等(2024)提出的无辅助损失负载均衡(ALF-LB)方法提供理论框架,将其建模为一种原始-对偶方法,每轮训练仅需一次常数时间更新来求解分配问题。在简化确定性设定下,框架揭示了三个关键性质:(i) 拉格朗日目标函数的单调改进条件,(ii) 将令牌从过载专家转移到欠载专家的偏好规则,(iii) 近似平衡保证。进一步引入训练中的随机性与动态性,采用广义在线优化形式,证明目标函数具备强凸性,从而在特定步长选择下获得对数期望遗憾界。此外,通过10亿参数的DeepSeekMoE模型真实实验验证理论结论。整体构建了分析s-MoE中无辅助损失负载均衡的原理性框架。

原文摘要 · Abstract (English)

In large-scale AI training, Sparse Mixture-of-Experts (s-MoE) layers enable scaling by activating only a small subset of experts per token. An operational challenge in this design is load balancing: routing tokens to minimize the number of idle experts, which is important for the efficient utilization of costly GPUs and for the thorough training of architecture parameters across all experts. We provide a theoretical framework for analyzing the Auxiliary-Loss-Free Load Balancing (ALF-LB) procedure -- proposed by DeepSeek's Wang et al. (2024) -- by casting it as a primal-dual method using a single-shot, constant-time update per training iteration for solving an assignment problem. First, in a stylized deterministic setting, our framework yields several insightful structural properties: (i) a monotonic improvement condition for the Lagrangian objective, (ii) a preference rule that moves tokens from overloaded to underloaded experts, and (iii) an approximate-balancing guarantee. Then, we incorporate the stochastic and dynamic nature of AI training using a generalized online optimization formulation. In the online setting, we derive a strong convexity property of the objective that leads to a logarithmic expected regret bound under certain step-size choices. Additionally, we present real experiments on 1B-parameter DeepSeekMoE models to complement our theoretical findings. Together, these results build a principled framework for analyzing the Auxiliary-Loss-Free Load Balancing of s-MoE in AI models.

专家模型负载均衡理论分析深度学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。