用门控熵实现视觉语言模型动态路由，提升资源利用效率
GeMoE: Gating Entropy is All You Need for Uncertainty-aware Adaptive Routing in MoE-based Large Vision-Language Models
- 将路由视为信息编码，用门控熵衡量输入复杂度
- 相比静态路由，性能保留99.5%，专家激活稀疏性提升36.5%
- 适合追求高效推理的MoE类大模型开发者
随着模型参数和训练数据的增长，大型视觉语言模型（LVLMs）的指令遵循与泛化能力显著提升。基于混合专家（MoE）架构，LVLMs在保持推理成本可控的前提下扩展了参数容量。然而，传统MoE采用固定的Top-k路由策略，未能根据输入变化自适应调整专家数量，导致资源利用率不足。本文将令牌路由视为信息编码任务，将动态路由建模为最小描述长度（MDL）问题。通过验证MDL与门控熵在MoE场景下的关联，提出基于门控熵的不确定性感知自适应路由（GeMoE）。不同于传统静态或启发式动态路由方法，GeMoE显式建模模型复杂度与性能之间的权衡。通过门控熵评估令牌复杂度，自适应决定每个令牌应调用的专家数量。在多种骨干网络和基准测试上，该方法相比原始静态路由平均性能保留率达99.5%，同时平均专家激活稀疏性提升36.5%。
原文摘要 · Abstract (English)
With the increase in model parameters and training data, the instruction following and generalization capabilities of Large VisionLanguage Models (LVLMs) have been significantly improved. Based on the Mixture of Experts (MoE) architecture, LVLMs expand their parameter capacity while maintaining the inference cost. However, traditional MoE methods employ a Top-k static routing strategy, which fails to account for variations in the input and adaptively select the number of experts, resulting in suboptimal resource utilization. In this paper, we propose viewing token routing as an information encoding task, framing dynamic routing as a Minimum Description Length (MDL) problem in encoding By validating the connection between MDL and gating entropy in the MoE scenario, we introduce Gating Entropy-based Uncertainty-aware Adaptive Routing (GeMoE) for MoE. Unlike traditional static or heuristic-based dynamic routing methods, GeMoE explicitly models the trade-off between model complexity and performance. By using gating entropy to assess the complexity of tokens, GeMoE adaptively determines the number of experts each token should engage. On a wide range of backbones and benchmarks, our method achieves 99.5% average performance retention compared to the original static routing, while improving average expert activation sparsity by 36.5%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。