无线联邦LoRA微调中,动态调整参数更新的稀疏度与带宽,提升通信效率。
Two Stage Wireless Federated LoRA Fine-Tuning with Sparsified Orthogonal Updates
- 分两阶段设计:先离线选低秩结构,再在线根据信道实时优化稀疏度与带宽
- 提出SOFT方法,通过近正交性指导参数上传,避免计算完整矩阵乘积
- 理论证明收敛性,并基于李雅普诺夫优化实现联合优化,适合资源受限场景
联邦微调中的低秩适配(LoRA)仅传输两个低秩矩阵而非全模型,但现有方法通常预先固定LoRA秩作为人工调参。在无线网络中,秩既决定适应能力又影响上行负载,而可交付负载随信道衰落变化。为解决这一耦合问题,我们将无线联邦LoRA微调建模为双时间尺度设计:将预先根据统计信道信息选择的共享LoRA结构秩(离线优化秩)与每轮迭代中根据瞬时信道状态信息(CSI)自适应的稀疏化和带宽决策分离。针对每轮自适应,提出稀疏正交微调(SOFT),通过促进秩分量间的近正交性,使列向量与行向量范数乘积逼近分量的奇异值。该无需SVD的评分机制指导逐分量负载分配与组内条目选择,无需构造完整矩阵乘积。进一步推导出连接两时间尺度的收敛界,并开发两阶段联邦算法(TSFA),离线选择最优秩,线上通过李雅普诺夫优化联合优化稀疏化与带宽。
原文摘要 · Abstract (English)
Federated fine-tuning with low-rank adaptation (LoRA) communicates only two low-rank matrices instead of the full model, but existing methods typically fix the LoRA rank in advance as a manually tuned hyperparameter. In wireless networks, however, the rank determines both adaptation capacity and uplink payload, while the deliverable payload varies with the fading channel. To address this coupling, we formulate wireless federated LoRA fine-tuning as a two-timescale design that separates the \emph{offline-optimized rank}, i.e., the rank of the shared LoRA structure selected before training from statistical channel information, from the per-iteration sparsification and bandwidth decisions adapted to instantaneous CSI. For per-iteration adaptation, we propose sparsified orthogonal fine-tuning (\textbf{SOFT}), which promotes near-orthogonality among rank components so that the product of the corresponding column and row norms approximates each component's singular value. This SVD-free score guides component-wise payload allocation and within-component entry selection without forming the full matrix product. We further derive a convergence bound linking the two timescales and develop a two-stage federated algorithm (\textbf{TSFA}) that selects the offline-optimized rank offline and jointly optimizes sparsification and bandwidth online via Lyapunov optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。