提出动态跳过专家机制,让多模态大模型推理更快更准
MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping
- 用全局调制局部路由,精准判断每个令牌该用哪些专家
- 对视觉和语言模态分别设置阈值,跳过88%专家仍提升10.67%性能
- 无需重新训练,适配多模态模型,显著加速推理速度
多模态大语言模型(MLLMs)在视觉-语言任务中表现优异,但推理计算开销大。现有专家跳过方法原为单模态大模型设计,应用于多模态模型时导致性能显著下降,因其未考虑各专家层间贡献差异及跨模态令牌行为差异。为此,我们提出MoDES,首个无需训练的自适应专家跳过框架。它引入全局调制局部路由(GMLG)机制,将层级重要性融入局部路由概率,精确估计每令牌的专家重要性;再采用双模态阈值法(DMT),分模态处理以生成跳过策略。通过前沿搜索算法优化阈值,收敛时间从数日缩短至数小时。在13个基准上对3个模型系列的实验表明,当跳过88%专家(如Qwen3-VL-MoE-30B-A3B-Instruct)时,性能最高提升10.67%(97.33% vs. 86.66%),预填充时间提速2.16倍,解码时间提速1.26倍。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) Multimodal large language models (MLLMs) excel at vision-language tasks, but they suffer from high computational inefficiency. To reduce inference overhead, expert skipping methods have been proposed to deactivate redundant experts based on the current input tokens. However, we find that applying these methods-originally designed for unimodal large language models (LLMs)-to MLLMs results in considerable performance degradation. This is primarily because such methods fail to account for the heterogeneous contributions of experts across MoE layers and modality-specific behaviors of tokens within these layers. Motivated by these findings, we propose MoDES, the first training-free framework that adaptively skips experts to enable efficient and accurate MoE MLLM inference. It incorporates a globally-modulated local gating (GMLG) mechanism that integrates global layer-wise importance into local routing probabilities to accurately estimate per-token expert importance. A dual-modality thresholding (DMT) method is then applied, which processes tokens from each modality separately, to derive the skipping schedule. To set the optimal thresholds, we introduce a frontier search algorithm that exploits monotonicity properties, cutting convergence time from several days to a few hours. Extensive experiments for 3 model series across 13 benchmarks demonstrate that MoDES far outperforms previous approaches. For instance, when skipping 88% experts for Qwen3-VL-MoE-30B-A3B-Instruct, the performance boost is up to 10.67% (97.33% vs. 86.66%). Furthermore, MoDES significantly enhances inference speed, improving the prefilling time by 2.16$\times$ and the decoding time by 1.26$\times$. Our code is available at https://github.com/ModelTC/MoDES.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。