提出新方法保护微调大模型隐私，几乎不损失性能。
Provably Protecting Fine-Tuned LLMs from Training Data Extraction while Preserving Utility
- 只保留关键词的概率变化，其余大幅平滑以保隐私
- 理论保护能力比现有方法强数个数量级
- 适合需要高隐私保障的敏感数据微调场景
在敏感数据上微调大语言模型会引发训练数据提取（TDE）攻击，导致机密信息泄露。现有防御方法或缺乏严格隐私保证，或造成显著性能下降。我们发现微调会导致广泛概率变化，但仅需保留少数关键词的偏差即可；其余低影响变化可被大幅平滑且对性能影响极小。受此启发，我们提出SCP-Δ_r，一种基于近无访问自由（NAF）的算法，通过相对概率操作并利用基础模型平滑低影响词。该方法在理论上提供远超现有NAF方法的数量级优势，并在实验中实现强抗TDE攻击能力，同时保持极小性能损失。
原文摘要 · Abstract (English)
Fine-tuning large language models (LLMs) on sensitive datasets raises privacy concerns, as training data extraction (TDE) attacks can expose highly confidential information. Existing defenses against such attacks either lack formal privacy guarantees or incur substantial utility degradation. We observe that fine-tuning induces widespread probability shifts, yet preserving only a small subset of influential token-level deviations is sufficient; the remaining shifts can be aggressively smoothed with minimal impact on utility. Motivated by this insight, we propose SCP-$Δ_r$, a Near Access Freeness (NAF)-based algorithm that operates on relative probabilities and explicitly smooths low-impact tokens using a base model. SCP-$Δ_r$ achieves orders-of-magnitude better theoretical bounds than existing NAF based methods and provides strong empirical protection against TDE attacks with minimal performance loss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。