让普通设备也能高效微调大型专家混合模型，提升训练速度。
Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices
- 通过轻量级量化分析估算专家激活情况，降低计算开销。
- 动态合并专家并调整角色分配，兼顾效率与精度。
- 适合资源有限的设备参与大模型联邦微调，如消费级显卡。
基于专家混合（Mixture-of-Experts, MoE）的大语言模型在联邦微调中面临巨大计算需求与参与者资源受限的矛盾。现有方法依赖模型量化、计算卸载或专家剪枝，但因不切实际的系统假设及忽略MoE特性，难以达到理想性能。本文提出FLUX系统，支持在资源受限设备（如消费级GPU）上实现MoE-LMM的联邦微调，以最小化达到目标准确率所需时间。FLUX引入三项创新：(1) 基于量化本地分析，以极低开销估计专家激活；(2) 自适应层感知专家合并，减少资源消耗同时保持精度；(3) 采用探索-利用策略动态分配专家角色，平衡可微调与不可微调专家。在LLaMA-MoE和DeepSeek-MoE多个基准数据集上的实验表明，FLUX显著优于现有方法，时间到准确率最高提升4.75倍。
原文摘要 · Abstract (English)
Federated fine-tuning of Mixture-of-Experts (MoE)-based large language models (LLMs) is challenging due to their massive computational requirements and the resource constraints of participants. Existing working attempts to fill this gap through model quantization, computation offloading, or expert pruning. However, they cannot achieve desired performance due to impractical system assumptions and a lack of consideration for MoE-specific characteristics. In this paper, we propose FLUX, a system designed to enable federated fine-tuning of MoE-based LLMs across participants with constrained computing resources (e.g., consumer-grade GPUs), aiming to minimize time-to-accuracy. FLUX introduces three key innovations: (1) quantization-based local profiling to estimate expert activation with minimal overhead, (2) adaptive layer-aware expert merging to reduce resource consumption while preserving accuracy, and (3) dynamic expert role assignment using an exploration-exploitation strategy to balance tuning and non-tuning experts. Extensive experiments on LLaMA-MoE and DeepSeek-MoE with multiple benchmark datasets demonstrate that FLUX significantly outperforms existing methods, achieving up to 4.75X speedup in time-to-accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。