针对异构无线网络,提出动态切换LoRA的联邦微调方法。
A Federated Fine-Tuning Paradigm of Foundation Models in Heterogenous Wireless Networks
- 设备动态切换LoRA模块,缓解硬件差异与传输不稳影响。
- 在SST-2和QNLI数据集上提升测试准确率与能效。
- 适合资源受限的边缘智能场景,如移动设备协同学习。
边缘智能为移动设备提供低延迟、广覆盖服务成为可能。近年来,将低秩适应(LoRA)与联邦学习结合的微调机制推动了边缘智能发展。然而,在无线网络中,设备异构性及边缘设备的资源限制严重威胁联邦微调性能。为此,本文提出基于在线学习的异构无线网络联邦微调优化框架。首先,构建基于切换的联邦微调架构,边缘设备与基站协作,动态切换至LoRA模块以共同缓解设备异构性与传输不可靠问题。其次,通过理论分析导出推理风险差距的可计算上界;为提升泛化能力,建立带长期约束的非凸混合整数规划问题,并分解为模型切换、发射功率控制与带宽分配子问题。设计一种具有多项式复杂度的在线优化算法求解。最后,基于SST-2与QNLI数据集的仿真结果表明,该方法在测试准确率与能效方面均取得显著提升。
原文摘要 · Abstract (English)
Edge intelligence has emerged as a promising strategy to deliver low-latency and ubiquitous services for mobile devices. Recent advances in fine-tuning mechanisms of foundation models have enabled edge intelligence by integrating low-rank adaptation (LoRA) with federated learning. However, in wireless networks, the device heterogeneity and resource constraints on edge devices pose great threats to the performance of federated fine-tuning. To tackle these issues, we propose to optimize federated fine-tuning in heterogenous wireless networks via online learning. First, the framework of switching-based federated fine-tuning in wireless networks is provided. The edge devices switches to LoRA modules dynamically for federated fine-tuning with base station to jointly mitigate the impact of device heterogeneity and transmission unreliability. Second, a tractable upper bound on the inference risk gap is derived based on theoretical analysis. To improve the generalization capability, we formulate a non-convex mixed-integer programming problem with long-term constraints, and decouple it into model switching, transmit power control, and bandwidth allocation subproblems. An online optimization algorithm is developed to solve the problems with polynomial computational complexity. Finally, the simulation results on the SST-2 and QNLI data sets demonstrate the performance gains in test accuracy and energy efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。