提出轻量联邦微调方法，通信成本降10倍且适应数据异构。
Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models
- 在客户端嵌入张量化适配器，实现高效低通信微调。
- 在BERT和LLaMA上验证，通信量减少10倍，性能不下降。
- 适合隐私敏感场景下的大模型跨设备/跨机构微调。
参数高效微调（PEFT）通常假设大语言模型（LLMs）在单一设备数据上训练，但真实场景中数据常分布于多设备。联邦学习（FL）通过将敏感数据保留在本地，保障用户隐私。然而将PEFT融入FL面临通信开销大和数据异构两大挑战。本文提出FedTT与FedTT+，在客户端编码器/解码器块中集成张量化适配器，支持跨孤岛与大规模跨设备联邦学习。FedTT+通过自适应冻结部分张量因子，增强对数据异构的鲁棒性，进一步减少可训练参数。在BERT和LLaMA上的实验表明，所提方法有效应对数据异构问题，在通信成本降低高达10倍的同时，性能优于或持平现有联邦PEFT方法。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) methods typically assume that Large Language Models (LLMs) are trained on data from a single device or client. However, real-world scenarios often require fine-tuning these models on private data distributed across multiple devices. Federated Learning (FL) offers an appealing solution by preserving user privacy, as sensitive data remains on local devices during training. Nonetheless, integrating PEFT methods into FL introduces two main challenges: communication overhead and data heterogeneity. In this paper, we introduce FedTT and FedTT+, methods for adapting LLMs by integrating tensorized adapters into client-side models' encoder/decoder blocks. FedTT is versatile and can be applied to both cross-silo FL and large-scale cross-device FL. FedTT+, an extension of FedTT tailored for cross-silo FL, enhances robustness against data heterogeneity by adaptively freezing portions of tensor factors, further reducing the number of trainable parameters. Experiments on BERT and LLaMA models demonstrate that our proposed methods successfully address data heterogeneity challenges and perform on par or even better than existing federated PEFT approaches while achieving up to 10$\times$ reduction in communication cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。