无需训练数据即可清除大模型中的敏感信息
Data-Free Privacy-Preserving for LLMs via Model Inversion and Selective Unlearning
- 通过语言模型反演生成伪造敏感数据
- 在低秩适配空间中实现逐标记删除，准确率超90%
- 适合隐私敏感场景下模型的合规性改造
大型语言模型虽具强大能力，但可能记忆训练数据中的敏感个人身份信息（PII），引发严重隐私风险。现有机器遗忘技术多依赖训练数据，而实际部署中训练数据常为专有或不可获取。为此，我们提出无数据选择性遗忘（DFSU）框架，可在不访问训练数据的情况下从大模型中移除敏感PII。方法首先通过语言模型反演生成伪PII，构建对应标记级隐私掩码，并在低秩适配（LoRA）子空间中利用对比掩码损失实现标记级选择性遗忘。在AI4Privacy PII-Masking数据集上使用Pythia模型的实验表明，该方法能有效消除目标PII，同时保持模型性能。
原文摘要 · Abstract (English)
Large language models (LLMs) exhibit powerful capabilities but risk memorizing sensitive personally identifiable information (PII) from their training data, posing significant privacy concerns. While machine unlearning techniques aim to remove such data, they predominantly depend on access to the training data. This requirement is often impractical, as training data in real-world deployments is commonly proprietary or inaccessible. To address this limitation, we propose Data-Free Selective Unlearning (DFSU), a novel privacy-preserving framework that removes sensitive PII from an LLM without requiring its training data. Our approach first synthesizes pseudo-PII through language model inversion, then constructs token-level privacy masks for these synthetic samples, and finally performs token-level selective unlearning via a contrastive mask loss within a low-rank adaptation (LoRA) subspace. Extensive experiments on the AI4Privacy PII-Masking dataset using Pythia models demonstrate that our method effectively removes target PII while maintaining model utility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。