让复杂句子用更多专家，简单句子少用，提升大模型效率
Route Experts by Sequence, not by Token
- 把专家分配从逐词固定改为按序列动态分配
- 在高稀疏性下性能提升最高达16.9%
- 无需额外训练，兼容现有MoE模型
Mixture-of-Experts（MoE）架构通过每词仅激活部分专家来扩展大语言模型，但标准TopK路由对所有词分配相同数量的专家，忽略了其复杂度差异。已有自适应路由方法引入额外模块和超参数，常需重新训练。我们提出序列级TopK（SeqTopK），将专家预算从词级别转移至序列级别：在全部T个词中选取前T·K个专家，实现端到端学习的动态分配——复杂词获更多专家，简单词则更少，整体预算不变。该方法仅需少量代码修改，增加不到1%开销，完全兼容预训练MoE模型。在数学、编程、法律和写作任务上的实验表明，相比TopK及以往无参自适应方法，SeqTopK均有稳定提升，且在更高稀疏性下增益更显著（最高达16.9%）。结果表明，SeqTopK是一种简单、高效、可扩展的路由策略，尤其适合下一代极稀疏大模型。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) architectures scale large language models (LLMs) by activating only a subset of experts per token, but the standard TopK routing assigns the same fixed number of experts to all tokens, ignoring their varying complexity. Prior adaptive routing methods introduce additional modules and hyperparameters, often requiring costly retraining from scratch. We propose Sequence-level TopK (SeqTopK), a minimal modification that shifts the expert budget from the token level to the sequence level. By selecting the top $T \cdot K$ experts across all $T$ tokens, SeqTopK enables end-to-end learned dynamic allocation -- assigning more experts to difficult tokens and fewer to easy ones -- while preserving the same overall budget. SeqTopK requires only a few lines of code, adds less than 1% overhead, and remains fully compatible with pretrained MoE models. Experiments across math, coding, law, and writing show consistent improvements over TopK and prior parameter-free adaptive methods, with gains that become substantially larger under higher sparsity (up to 16.9%). These results highlight SeqTopK as a simple, efficient, and scalable routing strategy, particularly well-suited for the extreme sparsity regimes of next-generation LLMs. Code is available at https://github.com/Y-Research-SBU/SeqTopK.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。