FLAME让大模型在不同设备上高效微调，性能更优。
FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoE
- 用稀疏专家混合架构动态调整每台设备激活的专家数量。
- 在多种计算环境下均优于现有方法，性能提升显著。
- 适合资源差异大的分布式场景，如手机、边缘设备等。
现有的资源自适应LoRA联邦微调方法通过压缩全局LoRA矩阵，使客户端在不同算力条件下进行模型微调，但压缩会导致信息丢失，影响性能。为解决该问题，我们提出FLAME，一种基于稀疏专家混合（SMoE）架构的新型联邦学习框架。与以往方法不同，FLAME保留完整的全局LoRA矩阵，并通过调整每个客户端激活的专家数量实现端侧自适应。然而，将SMoE引入联邦学习带来新挑战：部分专家激活导致输出幅度不一致，以及各客户端间专家训练质量不均衡。FLAME通过轻量级重缩放机制和激活感知聚合策略有效应对这些挑战。在多种计算环境下的实验证明，FLAME始终优于现有方法，为资源自适应联邦学习提供了鲁棒且高效的解决方案。
原文摘要 · Abstract (English)
Existing resource-adaptive LoRA federated fine-tuning methods enable clients to fine-tune models using compressed versions of global LoRA matrices, in order to accommodate various compute resources across clients. This compression requirement will lead to suboptimal performance due to information loss. To address this, we propose FLAME, a novel federated learning framework based on the Sparse Mixture-of-Experts (SMoE) architecture. Unlike prior approaches, FLAME retains full (uncompressed) global LoRA matrices and achieves client-side adaptability by varying the number of activated experts per client. However, incorporating SMoE into federated learning introduces unique challenges, specifically, the mismatch in output magnitude from partial expert activation and the imbalance in expert training quality across clients. FLAME tackles these challenges through a lightweight rescaling mechanism and an activation-aware aggregation scheme. Empirical results across diverse computational settings demonstrate that FLAME consistently outperforms existing methods, providing a robust and effective solution for resource-adaptive federated learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。