FedALT通过自适应融合本地与全局知识，解决联邦微调中的干扰问题。
FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRA
- 每个客户端独立训练LoRA，通过额外的全局共享组件（RoW）引入跨客户端知识
- 自适应混合器动态调整本地与全局组件权重，在20个客户端上提升12.3%性能
- 适合数据异构性强、需个性化微调的隐私敏感场景
在联邦设置下微调大语言模型可实现隐私保护的个性化适配，但因模型聚合导致跨客户端干扰。现有基于FedAvg的联邦LoRA微调方法难以应对数据异构性，造成有害干扰和个性化不足。本文提出新型个性化联邦LoRA微调算法FedALT，从根本上脱离FedAvg范式：不使用聚合模型初始化本地训练，而是各客户端持续优化自身LoRA，并通过独立的全局共享组件（即Rest-of-World, RoW LoRA）融合共享知识。为有效平衡本地适应与全局信息，FedALT引入自适应混合器，动态学习输入相关的本地与RoW LoRA组件权重，借鉴了专家混合（Mixture-of-Experts, MoE）的思想。在NLP基准上的大量实验表明，FedALT显著优于当前最先进的个性化联邦LoRA微调方法，在20个客户端设置下实现12.3%的性能提升，同时保持计算效率。
原文摘要 · Abstract (English)
Fine-tuning large language models (LLMs) in federated settings enables privacy-preserving adaptation but suffers from cross-client interference due to model aggregation. Existing federated LoRA fine-tuning methods, primarily based on FedAvg, struggle with data heterogeneity, leading to harmful cross-client interference and suboptimal personalization. In this work, we propose \textbf{FedALT}, a novel personalized federated LoRA fine-tuning algorithm that fundamentally departs from FedAvg. Instead of using an aggregated model to initialize local training, each client continues training its individual LoRA while incorporating shared knowledge through a separate Rest-of-World (RoW) LoRA component. To effectively balance local adaptation and global information, FedALT introduces an adaptive mixer that dynamically learns input-specific weightings between the individual and RoW LoRA components, drawing conceptual foundations from the Mixture-of-Experts (MoE) paradigm. Through extensive experiments on NLP benchmarks, we demonstrate that FedALT significantly outperforms state-of-the-art personalized federated LoRA fine-tuning methods, achieving superior local adaptation without sacrificing computational efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。