针对视觉语言模型专家合并中忽略阶段差异的问题，提出按阶段感知的合并方法。
Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Models
- 基于各阶段归一化路由统计构建专家角色画像，区分图像、问题、回答阶段作用
- 在相同保留专家比例下，相比全局聚合方法性能提升最高达9.6%
- 适合需要轻量化部署但又需保持多阶段处理能力的视觉语言模型
混合专家视觉语言模型（MoE-VLM）通过稀疏专家激活提升容量，但部署需存储全部专家。训练无关的专家合并可减轻负担，现有基于路由的方法通常对所有标记全局聚合路由统计以判断合并兼容性。然而，MoE-VLM推理具有阶段结构：图像上下文标记携带视觉信息，问题标记定义查询，回答标记生成输出，三者数量与路由分布不同。由于图像上下文标记数量远超其他，全局聚合会过度强调其处理，掩盖阶段特异性专家角色，使服务于不同阶段的专家看似可互换，导致性能下降。因此，我们主张专家合并应保留阶段特异性角色，依据专家在不同阶段的表现而非全局路由统计判断兼容性。基于此，提出训练无关的RoleMerge方法：从阶段归一化路由统计构建每个专家的路由角色轮廓（RRP），捕捉其相对阶段偏好；根据专家-阶段信息损失，合并角色兼容的专家及其对应路由器条目，同时保留回答解码专家的区分性。在三个模型和多个基准上的实验表明，与替代方法相比，RoleMerge在匹配专家保留率下更有效保持原始模型性能，六任务宏平均性能相对提升最高达9.6%。结果验证了阶段条件专家角色比全局路由聚合更适合用于MoE-VLM专家合并。
原文摘要 · Abstract (English)
Mixture-of-experts vision-language models (MoE-VLMs) increase model capacity with sparse expert activation, yet deployment requires storing the full expert pool. Training-free expert merging reduces this burden, and many routing-based methods aggregate routing statistics across all tokens to determine merge compatibility. However, MoE-VLM inference is phase-structured: image-context tokens carry visual content, question tokens specify the query, and answer tokens produce the output, with different counts and routing distributions. Because image-context tokens are far more numerous, global aggregation can overemphasize image-context processing and obscure phase-conditioned expert roles, making experts serving different phases appear interchangeable and degrading model performance. We therefore argue that MoE-VLM expert merging should preserve phase-conditioned expert roles, judging compatibility by how experts serve different phases rather than globally aggregated routing statistics. Based on this view, we propose RoleMerge, a training-free method that constructs each expert's Routing Role Profile (RRP) from phase-normalized routing statistics, capturing its relative phase preference. Guided by expert-phase information loss, RoleMerge merges experts with compatible profiles and their corresponding router entries while preserving answer-decoding expert distinctions. Experiments on three models and multiple benchmarks show that RoleMerge preserves more of the full model's performance than alternative expert-merging methods at matched expert-retention ratios, with relative improvements of up to 9.6 percent in six-task macro-average performance. These results validate phase-conditioned expert roles as a more effective basis than global routing aggregation for MoE-VLM expert merging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。