针对多模态专家模型推理效率低的问题，提出自适应容量分配方案。
MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference
- 用信息熵加权评估视觉令牌语义价值，区分冗余与关键信息。
- 根据输入模态实时比例动态分配专家资源，避免资源浪费。
- 无需训练，可直接部署于现有多模态专家模型，提升推理效率。
多模态大语言模型的专家混合（MoE MLLMs）在专家并行（EP）推理中因慢节点效应面临显著效率瓶颈。这一问题在多模态场景下更严重：现有基于令牌数量的负载均衡方法无法解决两个独特挑战：（1）信息异质性，即大量冗余视觉令牌与语义关键令牌被同等对待；（2）模态动态性，即任务间视觉与文本比例变化导致资源错配。为此，我们提出MACS（模态感知容量扩展），一种无需训练的推理框架。具体而言，MACS引入熵加权负载机制量化视觉令牌的语义价值，缓解信息异质性；同时设计动态模态自适应容量机制，根据输入实时模态组成分配专家资源。大量实验表明，MACS在多个多模态基准上显著优于现有方法，为MoE MLLMs在EP推理中的高效部署提供了新颖且稳健的解决方案。
原文摘要 · Abstract (English)
Mixture-of-Experts Multimodal Large Language Models (MoE MLLMs) suffer from a significant efficiency bottleneck during Expert Parallelism (EP) inference due to the straggler effect. This issue is worsened in the multimodal context, as existing token-count-based load balancing methods fail to address two unique challenges: (1) Information Heterogeneity, where numerous redundant visual tokens are treated equally to semantically critical ones, and (2) Modality Dynamics, where varying visual to text ratios across tasks lead to resource misallocation. To address these challenges, we propose MACS (Modality-Aware Capacity Scaling), a training-free inference framework. Specifically, MACS introduces an Entropy-Weighted Load mechanism to quantify the semantic value of visual tokens, addressing information heterogeneity. Additionally, the Dynamic Modality-Adaptive Capacity mechanism allocates expert resources based on the real-time modal composition of the input. Extensive experiments demonstrate that MACS significantly outperforms existing methods on various multimodal benchmarks, providing a novel and robust solution for the efficient deployment of MoE MLLMs in EP inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。