解决联邦学习中专家模型冲突问题，提升大模型高效微调稳定性
Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Experts
- 通过重要性加权和梯度共识投影缓解客户端间专家优化冲突
- 在非独立同分布数据下收敛速度更快，全局与本地性能均提升
- 适合隐私敏感场景下的大规模语言模型协同训练
大语言模型持续扩展带来高昂计算成本，混合专家（MoE）通过稀疏激活成为高效微调的可扩展替代方案。联邦学习（FL）作为保护隐私的协作优化范式，但在数据异构条件下将引发专家优化冲突：不同客户端的数据分布导致相同索引的专家面对不一致甚至矛盾的特征-标签关联，聚合时产生破坏性干扰，使优化轨迹失稳、性能下降。为此，我们提出FC-MoE，一种面向MoE微调的联邦冲突感知框架。该框架采用重要性感知加权策略优先保留可靠本地更新，并通过梯度共识投影抑制冲突更新，保障全局优化路径稳定；同时引入本地知识保留机制，通过重新锚定领域特定残差，保护客户端专业能力。大量实验表明，FC-MoE在非独立同分布联邦环境中加速收敛，显著提升全局与本地模型性能。
原文摘要 · Abstract (English)
The continuous scaling of large language models (LLMs) incurs prohibitive computational costs, making Mixture-of-Experts (MoE) a scalable alternative for efficient fine-tuning via sparse activation. While federated learning (FL) emerges as the paradigm for privacy-preserving collaborative optimization, integrating MoE into FL under data heterogeneity may trigger conflicting expert optimizations. Client-specific data distributions force same-indexed experts to optimize under inconsistent or even conflicting feature-label correlations. This mismatch induces destructive interference during aggregation, thus destabilizing the optimization trajectory and degrading model performance. To address this issue, we propose FC-MoE, a federated conflict-aware framework for MoE fine-tuning. It employs an importance aware weighting scheme to prioritize reliable local updates and utilizes gradient consensus projection to suppress conflicting updates, ensuring a stable global optimization path. Moreover, a local knowledge retention mechanism further preserves specialized client expertise by re-anchoring domain-specific residuals. Extensive experiments demonstrate that FC-MoE accelerates convergence and enhances both global and local model performance in non-IID federated environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。