提出可扩展的专家混合架构优化框架,实现算力与模型配置的精准匹配。
Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization
- 构建三重联合约束:每令牌浮点数、活跃参数与总参数,统一评估标准
- 将16维搜索空间压缩为两阶段低维优化,跨六数量级算力验证有效
- 发现大模型规模下最优配置带宽变宽,利于实际部署灵活权衡
大型语言模型的缩放定律指导宏观资源配置,但将其转化为精确的专家混合(MoE)架构仍面临组合爆炸式设计空间难题。现有研究受限于实验预算,要么在缩放公式中添加额外的MoE变量,导致拟合不可靠;要么固定所有非MoE因素,忽略全局交互。本文提出一个可复用的全息式MoE架构优化框架,填补这一空白。首先指出每令牌浮点数(FLOPs per token)作为公平性度量对MoE模型不充分,因不同层类型计算密度差异可能在不增加计算成本的情况下膨胀参数;进而建立包含每令牌浮点数、活跃参数和总参数的联合约束三元组。通过代数约束与隐藏维度的保秩性质,将16维架构搜索空间缩减为两个顺序的低维阶段。在跨越六个数量级算力的数百个MoE模型上验证,该框架生成稳健的缩放定律,能将任意算力预算映射为完整且最优的MoE架构。关键发现是:随着模型规模增大,近优配置范围变宽,使从业者可量化权衡缩放建议与基础设施限制。
原文摘要 · Abstract (English)
Scaling laws for Large Language Models govern macroscopic resource allocation, yet translating them into precise Mixture-of-Experts (MoE) architectural configurations remains an open problem due to the combinatorially vast design space. Existing MoE scaling studies are constrained by experimental budgets to either augment scaling formulas with extra MoE variables, risking unreliable fits, or fix all non-MoE factors, ignoring global interactions. We propose a reusable framework for holistic MoE architectural optimization that bridges this gap. We first show that FLOPs per token alone is an inadequate fairness metric for MoE models because differing computational densities across layer types can inflate parameters without proportional compute cost, and establish a joint constraint triad of FLOPs per token, active parameters, and total parameters. We then reduce the 16-dimensional architectural search space to two sequential low-dimensional phases through algebraic constraints and a rank-preserving property of the hidden dimension. Validated across hundreds of MoE models spanning six orders of magnitude in compute, our framework yields robust scaling laws that map any compute budget to a complete, optimal MoE architecture. A key finding is that the near-optimal configuration band widens with scale, giving practitioners quantitative flexibility to balance scaling law recommendations against infrastructure constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。