用博弈论优化专家模型融合，让专家协作更平衡高效。
Expert Merging in Sparse Mixture of Experts with Nash Bargaining
- 引入纳什议价机制，动态分配专家权重，避免平均化缺陷。
- 在语言、图像等任务上超越现有方法，大模型下仍表现稳定。
- 适合追求高效专家协作的AI研究者与工业部署团队。
现有稀疏专家混合模型（SMoE）的专家融合策略多依赖输入相关或无关的参数平均，缺乏合理的加权机制。本文从博弈论视角重新审视专家融合，揭示专家间的合作与竞争关系。基于此提出纳什专家融合（NAMEx）框架，将纳什议价引入融合过程，实现更均衡高效的专家协作。同时，结合复杂动量机制，在理论上保证收敛性的同时加速专家传播。大量实验表明，NAMEx在语言建模、文本分类、图像分类及数据污染下的零样本鲁棒性任务中持续优于现有方法，并可无缝集成至主流MoE架构。进一步验证其在大规模系统中的可扩展性，成功应用于Qwen1.5-MoE（14B）和DeepSeek-MoE（16B），在零样本与微调场景均有效。代码已开源：https://github.com/anh147/NAMEx。
原文摘要 · Abstract (English)
Existing expert merging strategies for Sparse Mixture of Experts (SMoE) typically rely on input-dependent or input-independent averaging of expert parameters, but often lack a principled weighting mechanism. In this work, we reinterpret expert merging through the lens of game theory, revealing cooperative and competitive dynamics among experts. Based on this perspective, we introduce Nash Merging of Experts (NAMEx), a novel framework that incorporates Nash Bargaining into the merging process, enabling more balanced and efficient collaboration among experts. Additionally, we incorporate complex momentum into NAMEx to accelerate expert propagation with theoretical guarantees for convergence. Extensive experiments across language modelling, text classification, image classification, and zero-shot robustness under data corruption show that NAMEx consistently outperforms competing methods while integrating seamlessly with popular MoE architectures. Finally, we demonstrate NAMEx's scalability by applying it to large-scale systems, including Qwen1.5-MoE (14B) and DeepSeek-MoE (16B), where it proves effective in both zero-shot and fine-tuning settings. The code is publicly available at: https://github.com/anh147/NAMEx.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。