arXiv:2507.22565cs.LGcs.AI2025-07

用强化学习动态调整隐私保护,让大模型微调更高效更安全。

Efficient Differentially Private Fine-Tuning of LLMs via Reinforcement Learning

  • 用强化学习自动调节每层参数的梯度裁剪和噪声注入。
  • 在相同隐私约束下,模型性能提升5.6%,训练速度加快71%。
  • 适合关注隐私与效果平衡的AI研发和医疗领域应用者。

大语言模型在敏感数据(如医疗数据)上训练时,隐私与模型性能的矛盾日益突出。差分隐私随机梯度下降(DP-SGD)虽能保障形式化隐私,但强制梯度裁剪和加噪会显著降低样本效率和最终精度。现有方法均采用固定全局控制参数,无法适应优化过程中的动态变化,导致要么过度消耗隐私预算,要么模型性能不佳。本文提出RLDP,首个将差分隐私优化视为闭环控制问题并引入深度强化学习的方法。它实时感知学习动态,通过软动作价值(SAC)超策略在线选择细粒度的每参数梯度裁剪阈值和噪声强度,自主分配隐私预算。在GPT2-small、Llama-1B、Llama-3B和Mistral-7B上超过1600次消融实验表明,RLDP实现困惑度降低1.3%-30.5%(均值5.4%),下游任务性能平均提升5.6%;仅需13%-43%的梯度更新预算即可达到基线最终性能(平均提速71%),同时满足相同的(ε, δ)-DP约束,且对成员推断和可卡尼提取攻击的敏感性更低或相当。

原文摘要 · Abstract (English)

The tension between data privacy and model utility has become the defining bottleneck for the practical deployment of large language models (LLMs) trained on sensitive corpora including healthcare. Differentially private stochastic gradient descent (DP-SGD) guarantees formal privacy, yet it does so at a pronounced cost: gradients are forcibly clipped and perturbed with noise, degrading sample efficiency and final accuracy. Numerous variants have been proposed to soften this trade-off, but they all share a handicap: their control knobs are hard-coded, global, and oblivious to the evolving optimization landscape. Consequently, practitioners are forced either to over-spend privacy budget in pursuit of utility, or to accept mediocre models in order to stay within privacy constraints. We present RLDP, the first framework to cast DP optimization itself as a closed-loop control problem amenable to modern deep reinforcement learning (RL). RLDP continuously senses rich statistics of the learning dynamics and acts by selecting fine-grained per parameter gradient-clipping thresholds as well as the magnitude of injected Gaussian noise. A soft actor-critic (SAC) hyper-policy is trained online during language model fine-tuning; it learns, from scratch, how to allocate the privacy budget where it matters and when it matters. Across more than 1,600 ablation experiments on GPT2-small, Llama-1B, Llama-3B, and Mistral-7B, RLDP delivers perplexity reductions of 1.3-30.5% (mean 5.4%) and an average 5.6% downstream utility gain. RLDP reaches each baseline's final utility after only 13-43% of the gradient-update budget (mean speed-up 71%), all while honoring the same ($ε$, $δ$)-DP contract and exhibiting equal or lower susceptibility to membership-inference and canary-extraction attacks.

差分隐私强化学习大模型微调隐私安全

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。