自适应分配噪声，让语言模型在保护隐私的同时提升性能。
Fine-Tuning Language Models with Differential Privacy through Adaptive Noise Allocation
- 根据参数重要性动态分配噪声，而非统一加噪。
- 在多个数据集上缩小了普通微调与隐私保护微调的性能差距。
- 适合关注模型隐私与性能平衡的研究者和开发者。
语言模型能够记忆详细模式和信息，带来双重影响：利用存储知识在下游任务中表现优异，但同时也引发严重隐私问题。传统的基于差分隐私的训练方法通过在所有参数上采用均匀噪声分布提供稳健保护，但忽略了各参数在隐私保护中的敏感性和贡献差异，常导致模型性能不佳。为此，我们提出ANADP算法，基于模型参数的重要性自适应地分配添加噪声。实验表明，ANADP在一系列数据集上缩小了常规微调与传统差分隐私微调之间的性能差距，同时满足预期的隐私约束。
原文摘要 · Abstract (English)
Language models are capable of memorizing detailed patterns and information, leading to a double-edged effect: they achieve impressive modeling performance on downstream tasks with the stored knowledge but also raise significant privacy concerns. Traditional differential privacy based training approaches offer robust safeguards by employing a uniform noise distribution across all parameters. However, this overlooks the distinct sensitivities and contributions of individual parameters in privacy protection and often results in suboptimal models. To address these limitations, we propose ANADP, a novel algorithm that adaptively allocates additive noise based on the importance of model parameters. We demonstrate that ANADP narrows the performance gap between regular fine-tuning and traditional DP fine-tuning on a series of datasets while maintaining the required privacy constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。