联邦多任务微调让大模型在本地更准，全局不掉队
MIRA: A Method of Federated MultI-Task Learning for LaRge LAnguage Models
- 联邦框架下融合多任务学习，共享客户端任务与数据分布信息
- 使用LoRA降低参数量，本地损失更低，全局性能相当
- 适合数据异构性强、需保护隐私的模型微调场景
本文提出一种受多任务学习启发的联邦微调方法，用于大型语言模型（LLMs）。该方法利用各客户端模型结构，设计了一种考虑其他客户端任务与数据分布的学习机制。为缓解大模型常见的计算与通信开销，采用参数高效微调技术——低秩适应（LoRA），显著减少可训练参数数量。在多个数据集与模型上的实验表明，该方法在平均性能与本地性能方面均优于现有联邦微调框架。相比基线，本方案实现了更低的本地损失，同时保持相近的全局表现。
原文摘要 · Abstract (English)
In this paper, we introduce a method for fine-tuning Large Language Models (LLMs), inspired by Multi-Task learning in a federated manner. Our approach leverages the structure of each client's model and enables a learning scheme that considers other clients' tasks and data distribution. To mitigate the extensive computational and communication overhead often associated with LLMs, we utilize a parameter-efficient fine-tuning method, specifically Low-Rank Adaptation (LoRA), reducing the number of trainable parameters. Experimental results, with different datasets and models, demonstrate the proposed method's effectiveness compared to existing frameworks for federated fine-tuning of LLMs in terms of average and local performances. The proposed scheme outperforms existing baselines by achieving lower local loss for each client while maintaining comparable global performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。