解决异构客户端低秩微调的聚合难题，提升联邦学习效率
SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks
- 客户端用线性压缩传输更新，避免维度不匹配
- 在GLUE上微调RoBERTa-Large，性能超越现有方法
- 支持异构秩设置，适合资源不均的联邦场景
我们研究大型神经网络在低秩适应（LoRA）下的联邦参数高效微调。将LoRA与联邦微调结合带来新挑战：客户端可能采用不同LoRA秩，导致因子矩阵维度不兼容，且因子级平均存在双线性不匹配问题。为此提出SeFoRA，一种基于线性压缩的联邦LoRA算法，客户端传输本地更新的线性投影，使聚合可在服务器端直接进行。该方法缓解了双线性不匹配，并允许在全模型的小子空间中聚合。进一步提出秩同质版本SeFoRA-Ho，支持在此设定下直接适配器聚合。理论证明，在秩同质情况下，算法以$ ext{O}(1/T)$速率收敛至一阶驻点邻域。在GLUE数据集上微调RoBERTa-Large的数值实验表明，本方法显著优于现有最优方案。
原文摘要 · Abstract (English)
We consider federated parameter efficient fine-tuning of large neural networks with low-rank adaptation (LoRA,~Hu et al.\ 2022). Combining LoRA with federated PEFT introduces challenges absent from either setting alone: clients may use different LoRA ranks, making their factor matrices dimension-incompatible, and factor-wise averaging suffers from a bilinear mismatch. We propose SeFoRA, a sketch-aggregated federated LoRA algorithm in which each client transmits a linear sketch of its local updates, enabling direct aggregation at the federator. As a result, SeFoRA alleviates the bilinear mismatch, and allows for aggregation in a small subspace of the full model. We introduce a rank-homogeneous version called SeFoRA-Ho which allows for direct adapter aggregation in this setting. We prove convergence to a neighborhood of the first-order stationary point at rate $\cO(1/T)$ for the rank-homogeneous setting. Numerical experiments on fine-tuning RoBERTa-Large on GLUE datasets show how our algorithms outperform the state-of-the-art.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。