针对视觉语言模型中专家路由的分布差异,提出自适应路由机制提升性能。
Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Model
- 按模态差异设计路由策略,区分语言与视觉的分布特性。
- 视觉任务中激活更多专家,增强对稀有关键图像的处理能力。
- 在多类视觉语言和视觉基准上分别提升1.2%~2.1%,适合高效大模型优化。
混合专家(MoE)架构通过将密集网络替换为稀疏结构,在大型视觉语言模型(LVLMs)中受到广泛关注,可在激活更少参数的情况下实现相当的性能。现有LVLM的MoE方法主要聚焦于标记到专家的路由(TER),促使不同专家专注于特定标记的处理。然而,这些方法通常依赖负载均衡机制,忽视了视觉与语言模态间的固有分布差异。为此,本文提出长尾分布感知路由器(LTDR),解决两个关键挑战:(1)模态特异性分布感知路由。观察发现,语言的TER分布相对均匀,而视觉的TER呈现长尾分布。这一模态差异推动了针对不同模态设计专用路由策略。(2)视觉特异性动态专家激活。考虑到高信息量的视觉尾部标记的重要性,引入受数据增强启发的策略,增加激活专家数量,确保对这些罕见但关键标记的学习充分性。在视觉语言与视觉基准测试中,本方法均实现一致提升,在视觉语言任务上提升1.2%/2.1%,在视觉任务上提升1.6%。
原文摘要 · Abstract (English)
The mixture-of-experts (MoE) architecture, which replaces dense networks with sparse ones, has attracted significant attention in large vision-language models (LVLMs) for achieving comparable performance while activating far fewer parameters. Existing MoE architectures for LVLMs primarily focus on token-to-expert routing (TER), encouraging different experts to specialize in processing specific tokens. However, these methods typically rely on the load balancing mechanism, neglecting the inherent distributional differences between vision and language modalities. To address this limitation, we propose the Long-Tailed Distribution-aware Router (LTDR) for vision-language TER, which tackles two key challenges: (1) Modality-specific distribution-aware routing. We observe that language TER generally follows a relatively uniform distribution, whereas vision TER exhibits a long-tailed distribution. This modality discrepancy motivates the design of specialized routing strategies for each modality. (2) Vision-specific dynamic expert activation. Recognizing the importance of high-information vision tail tokens, we introduce a data-augmentation-inspired strategy that increases the number of activated experts, ensuring sufficient learning for these rare but informative tokens. On vision-language and vision benchmarks, our approach achieves consistent improvements, boosting performance by 1.2% / 2.1% on vision-language and 1.6% on vision benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。