提出动态专家路由机制，实现高效计算分配与负载均衡。
Expert Threshold Routing for Autoregressive Language Modeling with Dynamic Computation Allocation and Load Balancing
- 每个专家自维护阈值，按得分动态决定是否处理当前词元
- 在24亿参数规模下比传统方法低0.067交叉熵损失
- 无需额外损失函数，适合自回归语言建模场景
Token-choice Mixture-of-Experts（TC-MoE）将每个词元固定路由至有限数量的专家，限制了计算分配的灵活性，并需引入辅助损失以维持负载均衡。本文提出专家阈值（Expert Threshold, ET）路由机制，每个专家通过全局词元分布估算指数移动平均（EMA）阈值。在训练和推理阶段，若词元得分超过专家阈值，则独立选择该专家处理，实现动态计算分配并自然达成负载均衡，无需辅助损失。该完全因果机制不依赖批内其他词元，非常适合自回归语言建模。在24亿参数规模、基于FineWeb-Edu的数据集预训练实验中，ET相比TC-MoE降低0.067交叉熵损失，等效于使用1.6倍少的词元达到相同性能。
原文摘要 · Abstract (English)
Token-choice Mixture-of-Experts (TC-MoE) routes each token to a fixed number of experts, limiting dynamic computation allocation and requiring auxiliary losses to maintain load balance. We propose Expert Threshold (ET) routing, where each expert maintains an exponential moving average (EMA) threshold estimated from the global token distribution. At both training and inference, each token is independently routed to an expert if its score exceeds the expert's threshold, enabling dynamic computation allocation while achieving load balance without auxiliary losses. This fully causal mechanism eliminates dependence on other tokens in the batch, making it well-suited for autoregressive language modeling. In pretraining experiments scaling to 2.4B parameters on FineWeb-Edu, ET achieves 0.067 lower cross-entropy loss than TC-MoE, equivalent to reaching the same performance with 1.6$\times$ fewer tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。