针对异构设备的联邦微调，提出自适应低秩参数配置方法
Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devices
- 根据输出层附近增加高阶LoRA可节省资源
- 实测速度提升1.5至2.8倍，通信成本降低42.3%
- 适合资源受限的分布式语言模型微调场景
联邦微调（FedFT）被提出用于以分布式方式微调预训练语言模型。然而，在实际应用中存在资源限制与系统异构性两大挑战。现有方法依赖参数高效微调技术，如低秩适配（LoRA），但存在明显局限。基于FedFT特性，我们发现将高阶LoRA层靠近输出层添加可在保持性能的同时降低资源消耗。为此，提出新型基于LoRA的联邦微调框架LEGEND，解决LoRA层数（深度）与各层秩分布的配置难题。分析二者耦合关系，设计适用于异构设备的高效配置算法，提升微调效率。在由80台商用设备组成的物理平台上进行大量实验，结果表明，相较于先进方案，LEGEND在达到目标准确率时可实现1.5–2.8倍加速，并减少约42.3%通信开销。
原文摘要 · Abstract (English)
Federated fine-tuning (FedFT) has been proposed to fine-tune the pre-trained language models in a distributed manner. However, there are two critical challenges for efficient FedFT in practical applications, i.e., resource constraints and system heterogeneity. Existing works rely on parameter-efficient fine-tuning methods, e.g., low-rank adaptation (LoRA), but with major limitations. Herein, based on the inherent characteristics of FedFT, we observe that LoRA layers with higher ranks added close to the output help to save resource consumption while achieving comparable fine-tuning performance. Then we propose a novel LoRA-based FedFT framework, termed LEGEND, which faces the difficulty of determining the number of LoRA layers (called, LoRA depth) and the rank of each LoRA layer (called, rank distribution). We analyze the coupled relationship between LoRA depth and rank distribution, and design an efficient LoRA configuration algorithm for heterogeneous devices, thereby promoting fine-tuning efficiency. Extensive experiments are conducted on a physical platform with 80 commercial devices. The results show that LEGEND can achieve a speedup of 1.5-2.8$\times$ and save communication costs by about 42.3% when achieving the target accuracy, compared to the advanced solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。