通过切片路由提升Transformer扩展效率与平衡性
SliceMoE: Routing Embedding Slices Instead of Tokens for Fine-Grained and Balanced Transformer Scaling
- 将令牌隐藏向量切分为连续片段，按片段而非整令牌分配专家
- 推理速度比稠密模型快1.7倍，困惑度降低12%~18%
- 适合追求高效、均衡且可解释专家分工的NLP模型设计
混合专家（MoE）层通过将令牌路由到稀疏的前馈专家子集来扩展Transformer。然而，令牌级路由将完整语义空间分配给每个专家，导致容量瓶颈、负载不平衡及专长有限。我们提出SliceMoE，将d维嵌入向量划分为S个连续片段，对每个片段由轻量级共享路由器预测top-k专家。专家独立处理其分配的片段，输出重组后保持每令牌计算效率。由于不同令牌的片段在专家内交错，利用率更平滑。我们引入片段级容量损失、跨片段丢弃和高效的融合批处理GEMM核。在WikiText-103语言建模、WMT En-De翻译及三个文本分类数据集上的实验表明，SliceMoE相比稠密基线推理速度提升1.7倍，参数匹配的令牌级MoE困惑度降低12%~18%，并改善专家平衡性，实现对句法与语义子空间的可解释专长。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) layers scale transformers by routing tokens to a sparse subset of feed-forward experts. Token-level routing, however, assigns an entire semantic spectrum to each expert, creating capacity bottlenecks, load-balancing pathologies, and limited specialization. We introduce SliceMoE, an architecture that routes contiguous slices of a token's hidden vector. A d-dimensional embedding is partitioned into S slices, and for each slice, a lightweight shared router predicts the top-k experts. Experts operate on their assigned slices independently, and outputs are reassembled, maintaining per-token FLOP efficiency. Because slices from different tokens interleave within an expert, utilization is naturally smoother. We propose a slice-level capacity loss, cross-slice dropout, and efficient fused batched GEMM kernels. Experiments on WikiText-103 language modeling, WMT En-De translation, and three text-classification datasets show SliceMoE attains up to 1.7x faster inference than dense baselines, 12 to 18 percent lower perplexity than parameter-matched token-MoE, and improved expert balance, with interpretable expertise over syntactic versus semantic subspaces.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。