让扩散语言模型的专家计算更智能,按需求分配算力
REFLEX: Rethinking MoE Inference as Refinement-Aware Compute Allocation in Diffusion Language Models
- 根据每个词元的修复程度动态分配专家算力
- 平均减少15%专家计算量,生成质量不降反升
- 无需训练,适合现有扩散语言模型直接部署
混合专家(MoE)模型通过仅激活部分专家来提升参数容量,实现条件计算。然而,在扩散语言模型(DLMs)中,每次去噪前向会同时重访所有词元位置,尽管它们的修复需求差异显著,而默认固定路由分配相同的专家预算,造成算力与修复需求不匹配。本文提出REFLEX(Refinement-aware Flexible expert allocation),将MoE推理视为对异质词元修复状态的感知算力分配。该方法在不改变原有路由器的前提下,引入粗到细的专家预算层级结构,并利用前沿进展分数确定活跃块优先级,使计算与修复阶段对齐。在两个代表性MoE-DLM模型LLaDA-MoE和LLaDA2.0-mini上,多个基准测试显示,REFLEX平均降低15%的专家计算量,同时在多数任务上保持或提升生成质量。相比自回归式可变专家路由方法,其质量-算力权衡更稳定,验证了按修复需求分配算力的重要性。
原文摘要 · Abstract (English)
Mixture-of-experts (MoE) models increase parameter capacity by activating only a small subset of experts for each token. This conditional-computation paradigm has enabled autoregressive language models to scale model capacity without a proportional increase in per-token computation. In diffusion language models (DLMs), however, each denoising forward jointly revisits all token positions despite their sharply different refinement demands, while the default fixed token-choice routing assigns them a uniform expert budget, creating a mismatch between expert computation and refinement demand. We argue that MoE inference in DLMs should therefore be viewed as refinement-aware compute allocation across heterogeneous token refinement states. We propose REFLEX (\textbf{RE}finement-aware \textbf{FLEX}ible expert allocation), a training-free method that keeps the default router unchanged while reorganizing expert computation around the evolving refinement process. Specifically, REFLEX introduces a coarse-to-fine hierarchy for expert-budget allocation that aligns computation with block-relative refinement roles while using the Frontier-Progress Score to resolve active-block priorities. Across multiple widely used benchmarks on two representative MoE-based DLMs, LLaDA-MoE and LLaDA2.0-mini, REFLEX reduces allocated expert computation by 15\% on average while preserving or even improving generation quality on most benchmarks relative to default routing. Compared with autoregressive-style variable-expert routing methods, REFLEX also yields a more consistent quality--computation trade-off, further supporting the importance of allocating expert computation according to the heterogeneous refinement demands exposed within each denoising forward.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。