用隐私保护方法微调模型，防止数据泄露
An exponential mechanism based on quadratic approximations for fine-tuning machine learning models with privacy guarantees
- 基于二次近似构建效用函数，实现差分隐私的随机微调
- 可精确采样多元正态分布，理论保证隐私与精度
- 结合随机投影提升高维模型效率，适合医疗等敏感数据
微调将预训练模型适配到小规模敏感数据集，但存在记忆个别数据点的风险，使模型易受攻击者提取敏感信息。本文提出一种基于指数机制的随机化算法，在微调过程中确保差分隐私。核心思想是构建一个效用函数，融合预训练模型的局部二次近似与新数据信息。所得指数机制可闭式精确采样多元正态分布。本文建立了理论隐私保证、敏感度界和准确率估计。进一步引入随机投影策略，使方法适用于高维模型。在MNIST基准和MIMIC临床数据集上的数值实验表明，该方法性能优于现有差分隐私微调技术。
原文摘要 · Abstract (English)
Fine-tuning adapts a pretrained machine learning model to a small, sensitive dataset, but this process risks memorizing individual new data points, making the model vulnerable to adversaries who seek to extract sensitive information. In this work, we develop a randomized algorithm based on the exponential mechanism for fine-tuning while ensuring differential privacy. Our key idea is to construct a simple utility function that combines a local quadratic approximation of the pretrained model with information from the new dataset. The resulting exponential mechanism admits exact sampling from a multivariate normal distribution in closed form. We establish theoretical privacy guarantees, sensitivity bounds, and accuracy estimations for our method. We further introduce a random-projection strategy that makes the approach scalable to high-dimensional models. Numerical experiments on the MNIST benchmark and the MIMIC clinical dataset demonstrate competitive performance against existing differentially private fine-tuning techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。