提出轻量自适应剪裁方法，提升隐私保护下的语言模型联邦微调性能。
DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models
- 先用私有直方图估计初始剪裁阈值，再训练中自适应调整
- 相比现有方法平均提升6.6%准确率，且不额外消耗隐私预算
- 无需调参，适合资源受限的边缘设备上的隐私保护训练
联邦学习（FL）可在边缘设备上协作训练大规模语言模型，同时将用户数据保留在本地。然而，客户端提供的梯度仍可能泄露敏感信息。差分隐私随机梯度下降（DP-SGD）通过将每个客户端贡献剪裁至阈值 $C$ 并添加与 $C$ 成比例的噪声来缓解风险。现有自适应剪裁方法虽能动态调整 $C$，但需繁琐超参数调优，易侵蚀隐私预算。本文提出 DP-LAC，首先利用私有直方图估计法在数量级内逼近最优剪裁阈值，随后在训练过程中自适应调整该阈值，不消耗额外隐私预算，也无需引入新超参数。实验表明，DP-LAC 超过当前最佳自适应剪裁方法及原始 DP-SGD，平均准确率提升 6.6%。
原文摘要 · Abstract (English)
Federated learning (FL) enables the collaborative training of large-scale language models (LLMs) across edge devices while keeping user data on-device. However, FL still exposes sensitive information through client-provided gradients. Differentially private stochastic gradient descent (DP-SGD) mitigates this risk by clipping each client's contribution to a threshold $C$ and adding noise proportional to $C$. Existing adaptive clipping techniques dynamically adjust $C$ but demand tedious hyperparameter tuning, which can erode the privacy budget. In this paper, we introduce DP-LAC, a method that first estimates an initial clipping threshold within an order of magnitude of the optimum using private histogram estimation, and then adapts this threshold during training without consuming additional privacy budget or introducing new hyperparameters. Empirical results show that DP-LAC outperforms both state-of-the-art adaptive clipping methods and vanilla DP-SGD, achieving an average accuracy gain of $6.6\%$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。