针对资源受限设备，提出高效联邦微调小模型的新方法。
Efficient Federated Finetuning of Tiny Transformers with Resource-Constrained Devices
- 设计新型层微调方案，适配跨设备联邦学习
- 在有限内存和算力下，准确率显著优于现有方法
- 适合边缘设备部署，兼顾通信效率与性能
近年来，基于Transformer结构的大语言模型（LLMs）在众多机器学习任务中占据主导地位，尤其在文本处理方面表现突出。然而，这些模型训练需要大量数据，并带来高资源需求，尤其是在浮点运算量（FLOPs）和内存占用方面。为实现参数高效的微调，已发展出如Adapter或LoRA等技术。但观察发现，在联邦学习（FL）中使用LoRA时，虽保持参数效率，却存在内存和计算效率低的问题。为此，本文提出一种新型层微调方案，使跨设备联邦学习中的设备可在满足资源约束的前提下，利用预训练神经网络（NNs）。实验表明，该方案在同质或异构的计算与内存约束下均优于当前最先进方法，且在通信受限场景下与LoRA相当，实现了更高的联邦训练准确率。
原文摘要 · Abstract (English)
In recent years, Large Language Models (LLMs) through Transformer structures have dominated many machine learning tasks, especially text processing. However, these models require massive amounts of data for training and induce high resource requirements, particularly in terms of the large number of Floating Point Operations (FLOPs) and the high amounts of memory needed. To fine-tune such a model in a parameter-efficient way, techniques like Adapter or LoRA have been developed. However, we observe that the application of LoRA, when used in federated learning (FL), while still being parameter-efficient, is memory and FLOP inefficient. Based on that observation, we develop a novel layer finetuning scheme that allows devices in cross-device FL to make use of pretrained neural networks (NNs) while adhering to given resource constraints. We show that our presented scheme outperforms the current state of the art when dealing with homogeneous or heterogeneous computation and memory constraints and is on par with LoRA regarding limited communication, thereby achieving significantly higher accuracies in FL training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。