根据隐私敏感度动态加噪，提升大模型训练的隐私与性能平衡
SA-ADP: Sensitivity-Aware Adaptive Differential Privacy for Large Language Models
- 按个人数据敏感度分配噪声，而非统一加噪
- 在4个数据集上实现无损模型性能与强隐私保护
- 适合需兼顾隐私和模型效果的研究者与开发者
尽管大语言模型（LLMs）在下游任务中取得进展，其信息记忆能力引发了隐私担忧。如何在训练过程中保护个人身份信息（PII）仍是核心挑战。传统方法如差分隐私随机梯度下降（DP-SGD）通过均匀加噪提供可靠隐私保护，但牺牲了模型效用，造成隐私与性能的权衡。本文提出SA-ADP，一种敏感度感知的自适应差分隐私方法，根据个体PII的敏感度动态分配噪声。我们在四个数据集（ABCD、CUSTOMERSIM、Wikitext-2、UNSW-NB15）上评估该方法。结果表明，SA-ADP在保持强隐私保护的同时，模型性能接近无隐私保护基线（No-DP）和传统DP-SGD，未造成性能下降。
原文摘要 · Abstract (English)
Despite advances in the use of large language models (LLMs) in downstream tasks, their ability to memorize information has raised privacy concerns. Therefore, protecting personally identifiable information (PII) during LLM training remains a fundamental challenge. Conventional methods like Differential Privacy-Stochastic Gradient Descent (DP-SGD) provide robust privacy protection via uniform noising, protecting PII regardless of its distinct sensitivity. This comes at the expense of the model's utility, leading to a trade-off. In this paper, we propose SA-ADP, a sensitivity-aware approach that allocates noise based on the sensitivity of individual PII. We evaluated our method on four datasets (ABCD, CUSTOMERSIM, Wikitext-2, and UNSW-NB15 ). Our results show that SA-ADP achieves results comparable to the baseline (No-DP) and the conventional DP-SGD. This means that our method did not degrade the model's utility while still maintaining strong privacy protection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。