用可证明的融合方法,高效实现联邦学习中的个性化模型。
Communication-Efficient Personalized Adaptation via Federated-Local Model Merging
- 通过联邦与本地模型线性连接构造个性化模型
- 理论推导出最优融合权重,降低任务损失方差
- 在视觉语言任务上兼顾性能与通信效率
参数高效的微调方法(如LoRA)为适应大型视觉与语言模型提供了实用途径。但在联邦设置中存在任务异构性时,个性化面临通用知识与个性知识之间的平衡难题。现有方法多依赖启发式混合规则,缺乏理论支撑。此外,以往模型融合方法计算与通信开销大,难以在联邦环境中高效运行。本文提出Potara,一种基于理论指导的联邦个性化框架:为每个客户端构建个性化模型,融合两类互补模型——(i) 联邦模型(捕捉通用知识),(ii) 本地模型(捕捉个性化知识)。基于线性模式连通性,我们证明了期望任务损失存在方差迹上界,其最小化可导出闭式最优融合权重,确保融合模型性能优于联邦或本地任一模型。在视觉与语言基准上的实验表明,Potara持续提升个性化效果的同时显著减少通信开销,实现了优异的性能-通信权衡。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning methods, such as LoRA, offer a practical way to adapt large vision and language models to client tasks. However, this becomes particularly challenging under task-level heterogeneity in federated deployments. In this regime, personalization requires balancing general knowledge with personalized knowledge, yet existing approaches largely rely on heuristic mixing rules and lack theoretical justification. Moreover, prior model merging approaches are also computation and communication intensive, making the process inefficient in federated settings. In this work, we propose Potara, a principled framework for federated personalization that constructs a personalized model for each client by merging two complementary models: (i) a federated model capturing general knowledge, and (ii) a local model capturing personalized knowledge. Through the construct of linear mode connectivity, we show that the expected task loss admits a variance trace upper bound, whose minimization yields closed-form optimal mixing weights that guarantee a tighter bound for the merged model than for either the federated or local model alone. Experiments on vision and language benchmarks show that Potara consistently improves personalization while reducing communication, leading to a strong performance-communication trade-off.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。