让大模型在分散数据上更懂任务,自动区分不同任务并精准适配。
Task-Centric Personalized Federated Fine-Tuning of Language Models
- 按任务聚类而非按客户端,用适配器为每类任务定制模型。
- 在任务干扰下性能提升6.1%,跨任务泛化能力提高136%。
- 适合多任务、数据分布复杂的真实场景,如医疗或金融应用。
联邦学习(FL)在分布式私有数据上训练语言模型方面展现出巨大潜力,但对异构任务的模型聚合常导致整体性能下降。个性化联邦学习(pFL)旨在为每个客户端定制模型,虽提升了本地表现,却面临两大缺陷:(i) 泛化性差——当客户端需预测未见任务或数据分布发生变化时;(ii) 客户端内任务干扰——单个客户端数据包含多种分布,本地训练时易相互干扰。为此,我们提出 FedRouter,一种基于聚类的 pFL 方法,不按客户端个性化,而是为每类任务构建专用模型。通过两种聚类机制实现:局部聚类将适配器与任务样本关联,全局聚类将不同客户端中相似的适配器合并,形成以任务为中心的个性化模型。此外,引入评估路由机制,根据聚类结果将测试样本分配至最优适配器。在多任务数据集上的实验表明,与现有方法相比,FedRouter 在任务干扰场景下相对性能提升最高达6.1%,在泛化评估中相对提升高达136%。
原文摘要 · Abstract (English)
Federated Learning (FL) has emerged as a promising technique for training language models on distributed and private datasets of diverse tasks. However, aggregating models trained on heterogeneous tasks often degrades the overall performance of individual clients. To address this issue, Personalized FL (pFL) aims to create models tailored for each client's data distribution. Although these approaches improve local performance, they usually lack robustness in two aspects: (i) generalization: when clients must make predictions on unseen tasks, or face changes in their data distributions, and (ii) intra-client tasks interference: when a single client's data contains multiple distributions that may interfere with each other during local training. To tackle these two challenges, we propose FedRouter, a clustering-based pFL that builds specialized models for each task rather than for each client. FedRouter uses adapters to personalize models by employing two clustering mechanisms to associate adapters with specific tasks. A local clustering that associate adapters with task data samples and a global one that associates similar adapters from different clients to construct task-centric personalized models. Additionally, we propose an evaluation router mechanism that routes test samples to the best adapter based on the created clusters. Experiments comparing our method with existing approaches across a multitask dataset, FedRouter demonstrate strong resilience in these challenging scenarios performing up to 6.1% relatively better under tasks interference and up to 136% relative improvement under generalization evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。