用专家路由信息分组客户端,让联邦微调更精准高效。
MoE Router-Guided Clustering for Heterogeneous Federated Instruction Tuning

- 根据专家激活模式分组客户端,指导协作策略。
- 相比传统方法,在多个任务上提升个性化性能12%以上。
- 适合隐私敏感、数据异构的分布式大模型微调场景。
联邦指令微调使大语言模型能在不共享数据的前提下适应分散的私密数据。近期的混合专家(MoE)模型因其稀疏激活特性,在联邦学习中极具吸引力,可在扩展模型容量的同时降低计算与通信开销。然而,现有联邦MoE方法多聚焦于参数聚合与个性化,忽视了MoE模型路由行为所蕴含的协作信息。在指令分布异构的情况下,盲目聚合会导致负迁移,亟需识别哪些客户端应参与协同优化。本文提出ClientMorpher框架,利用预训练MoE模型的路由签名,在聚合前引导客户端协作。研究两种互补聚类策略:ClientMorpher-C直接基于专家激活特征聚类客户端;ClientMorpher-E先按跨客户端使用模式聚类专家,再推导客户端协作组。在Databricks Dolly-15K数据集上,采用病态及狄利克雷分布的异构客户端设置,评估多个指令遵循任务。实验表明,路由感知协作显著优于传统联邦平均与本地训练,且通信成本不变。结果验证了客户中心与专家中心聚类在稀疏MoE大模型联邦微调中的有效性与可扩展性。
原文摘要 · Abstract (English)
Federated instruction fine-tuning enables Large Language Models (LLMs) to adapt to decentralized, privacy-sensitive data without requiring data sharing. Recent Mixture-of-Experts (MoE) LLMs are particularly attractive for federated learning because their sparse activation reduces computation and communication while scaling model capacity. However, existing federated MoE methods primarily focus on parameter aggregation and personalization, overlooking the routing behavior of MoE models as a source of information for client collaboration. Under heterogeneous instruction distributions, indiscriminate aggregation can lead to negative transfer, highlighting the need to identify which clients should collaborate during federated optimization. We propose ClientMorpher, a routing-aware, personalized federated instruction fine-tuning framework that leverages routing signatures from pretrained MoE models to organize client collaboration prior to aggregation. We investigate two complementary clustering strategies: ClientMorpher-C, which directly clusters clients using expert activation profiles, and ClientMorpher-E, which first clusters experts based on their cross-client usage signatures and then derives client collaboration groups. We evaluate ClientMorpher for federated instruction fine-tuning on the Databricks Dolly-15K dataset, using pathological and Dirichlet-based heterogeneous client distributions across multiple instruction-following tasks. Experimental results show that routing-aware collaboration consistently improves personalized performance compared to conventional federated averaging and local training, while maintaining the same communication cost. Furthermore, our study shows that client-centric and expert-centric clustering provides an effective and scalable approach for personalized federated instruction fine-tuning of sparse MoE LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。