arXiv:2605.05769cs.LGcs.AI2026-05

自适应选择LoRA组件,提升隐私保护联邦学习的精度与稳定性。

Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learning

论文配图:Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learning
图 1 · 摘自论文原文
  • 按层和轮次动态选择活跃的LoRA组件,避免固定更新策略。
  • 在严格隐私预算下,GLUE任务最高提升7.5个百分点,MNLI-mm提升12.5个百分点。
  • 无需额外隐私开销,聚合成本仅为基线方法的1/33至1/180。

使用LoRA进行差分隐私联邦微调的大模型面临由LoRA乘法结构引发的聚合误差,该误差在DP噪声作用下被放大,导致稳定性和准确率下降。现有方法对所有层和通信轮次采用单一更新模式(或固定周期交替),忽略了两个LoRA因子间的结构差异及训练过程中的轮次动态变化。本文提出AS-LoRA,基于三个维度的自适应框架:(i) 层级自由度,每层独立选择激活组件;(ii) 轮次自适应,选择策略随通信轮次更新;(iii) 基于损失二阶近似的曲率感知评分。理论上,AS-LoRA消除了层绑定调度的重建误差下限,加速收敛,隐式引导解趋向更平坦的极小值,且不增加隐私成本。在严格DP预算与非独立同分布划分下,跨GLUE、SQuAD、CIFAR-100和Tiny-ImageNet数据集测试中,相较于联邦LoRA基线,AS-LoRA在GLUE上最高提升+7.5个百分点,在MNLI-mm上提升+12.5个百分点;同时在33–180倍更低的聚合成本下达到或超越SVD聚合方法,通信开销可忽略。代码已公开于https://anonymous.4open.science/r/as_lora-F75F/。

原文摘要 · Abstract (English)

Differentially private federated fine-tuning of large models with LoRA suffers from aggregation error caused by LoRA's multiplicative structure, which is further amplified by DP noise and degrades both stability and accuracy. Existing remedies apply a single update mode uniformly across all layers and all communication rounds (or alternate them on a fixed schedule), ignoring both the structural asymmetry between the two LoRA factors and the round-wise dynamics of training. We propose AS-LoRA, an adaptive framework defined by three axes (i) layer-wise freedom, in which each layer independently selects its active component, (ii) round-wise adaptivity, in which the selection updates over communication rounds, and (iii) a curvature-aware score derived from a second-order approximation of the loss. Theoretically, AS-LoRA eliminates the reconstruction-error floor of layer-tied schedules, accelerates convergence, implicitly biases solutions toward flatter minima, and incurs no additional privacy cost. Across GLUE, SQuAD, CIFAR-100, and Tiny-ImageNet under strict DP budgets and non-IID partitions, AS-LoRA improves over the federated LoRA baselines by up to $+7.5$ pp on GLUE and $+12.5$ pp on MNLI-mm for example, while matching or exceeding SVD-based aggregation methods at $33\text{--}180 \times$ lower aggregation cost and with negligible communication overhead. Code for the proposed method is available at https://anonymous.4open.science/r/as_lora-F75F/.

联邦学习隐私保护LoRA自适应优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。