根据语义重要性动态分配专家数量,提升多模态大模型效率。
AnyExperts: On-Demand Expert Allocation for Multimodal Language Models with Mixture of Expert
- 按语义重要性动态调整每令牌的专家数量,灵活分配算力。
- 图像视频任务少40%真实专家激活,文本密集任务减少10%使用量。
- 支持预算可控,虚拟专家占比不超过20%,适合资源受限场景。
多模态混合专家(MoE)模型为可扩展、高效的视觉语言系统提供了可行路径。然而,现有方法依赖固定的路由策略(通常每令牌激活固定数量的专家),忽略了跨模态语义重要性的固有差异,导致冗余令牌与关键令牌消耗相同资源,造成算力浪费。为此,我们提出 AnyExperts,一种新的按需、预算感知的动态路由框架,根据语义重要性动态分配每令牌的专家槽位数量,且总槽位数受固定范围约束。每个槽位由真实专家或虚拟专家填充,虚拟专家占比上限设为20%。模型自适应调节真实与虚拟专家比例:语义丰富区域调用更多真实专家,冗余内容则更多依赖虚拟专家。在视觉理解、音频理解及NLP理解等多样任务上评估,AnyExperts 在相同计算预算下提升性能。尤其在通用图像/视频任务中,实现相当精度的同时减少40%的真实专家激活;在文本密集任务(如OCR和NLP)中,保持性能并降低真实专家使用率10%。结果表明,细粒度、基于重要性的专家分配显著提升了多模态MoE模型的效率与有效性。
原文摘要 · Abstract (English)
Multimodal Mixture-of-Experts (MoE) models offer a promising path toward scalable and efficient large vision-language systems. However, existing approaches rely on rigid routing strategies (typically activating a fixed number of experts per token) ignoring the inherent heterogeneity in semantic importance across modalities. This leads to suboptimal compute allocation, where redundant tokens consume as many resources as critical ones. To address this, we propose AnyExperts, a novel on-demand, budget-aware dynamic routing framework that allocates a variable total number of expert slots per token based on its semantic importance. Crucially, to prevent uncontrolled compute growth, the total slots per token are constrained within a fixed range, and each slot is filled by either a real expert or a virtual expert, with the virtual share capped at a small maximum (e.g., 20%). The model then adaptively balances the real-to-virtual ratio per token, assigning more real experts to semantically rich regions and relying more on virtual experts for redundant content. Evaluated across diverse tasks in visual understanding, audio understanding, and NLP understanding, AnyExperts improves performance under the same compute budget. Notably, on general image/video tasks, it achieves comparable accuracy with 40% fewer real expert activations; on text-dense tasks (OCR and NLP), it maintains performance while reducing real expert usage by 10%. These results demonstrate that fine-grained, importance-driven expert allocation significantly enhances both the efficiency and effectiveness of multimodal MoE models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。