arXiv:2602.00879cs.LG2026-02被引 2

提出动态专家共享技术,解决扩散模型并行推理时专家爆炸问题。

Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMs

  • 通过序列级核心专家选择替代逐令牌优化,提升专家复用率。
  • 专家激活量减少超55%,延迟降低最高38%,精度损失仅1%。
  • 适合高并发文本生成场景,尤其对资源受限部署有重要意义。

在并行解码范式中,扩散大语言模型(dLLMs)因其生成质量与吞吐量的平衡而成为有前景的选择。然而,其与混合专家(MoE)架构的结合受限于专家爆炸问题:并行生成的标记数增加时,被激活的独立专家数量近似线性增长,导致显著的内存流量,使推理进入内存瓶颈状态,抵消了MoE和并行解码带来的效率优势。为此,我们提出动态专家共享(DES),将MoE优化从令牌级剪枝和传统专家跳过方法转向序列级核心集选择。为最大化专家复用,DES识别一个紧凑且高价值的专家集合,以满足整个并行解码块的需求。我们引入两种创新选择策略:(1) 序列内共享(DES-Seq),在序列层面自适应最优分配;(2) 显著性感知投票(DES-Vote),一种新机制,让标记基于聚合路由权重共同选举核心集。在MoE dLLMs上的大量实验表明,DES将唯一专家激活量减少超过55%,延迟最高降低38%,同时保持99%的原始精度,有效解耦了内存开销与并行度之间的关系。

原文摘要 · Abstract (English)

Among parallel decoding paradigms, diffusion large language models (dLLMs) have emerged as a promising candidate that balances generation quality and throughput. However, their integration with Mixture-of-Experts (MoE) architectures is constrained by an expert explosion: as the number of tokens generated in parallel increases, the number of distinct experts activated grows nearly linearly. This results in substantial memory traffic that pushes inference into a memory-bound regime, negating the efficiency gains of both MoE and parallel decoding. To address this challenge, we propose Dynamic Expert Sharing (DES), a novel technique that shifts MoE optimization from token-centric pruning and conventional expert skipping methods to sequence-level coreset selection. To maximize expert reuse, DES identifies a compact, high-utility set of experts to satisfy the requirements of an entire parallel decoding block. We introduce two innovative selection strategies: (1) Intra-Sequence Sharing (DES-Seq), which adapts optimal allocation to the sequence level, and (2) Saliency-Aware Voting (DES-Vote), a novel mechanism that allows tokens to collectively elect a coreset based on aggregated router weights. Extensive experiments on MoE dLLMs demonstrate that DES reduces unique expert activations by over 55% and latency by up to 38%, while retaining 99% of vanilla accuracy, effectively decoupling memory overhead from the degree of parallelism.

扩散模型MoE并行推理专家共享

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。