用少量参数微调大模型，既能提速又能降低隐私泄露风险
Efficient and Private: Memorisation under differentially private parameter-efficient fine-tuning in language models
- 用参数高效微调方法，在保护隐私前提下减少计算开销
- 相比全参数微调，同等性能下隐私泄露更少
- 适合关注隐私安全与资源效率的AI研发人员
为特定任务微调大语言模型会引入隐私风险，模型可能无意中记忆并泄露敏感训练数据。虽然差分隐私（DP）可缓解此类风险，但传统微调方法带来显著计算和性能代价。现有工作多聚焦全参数更新，计算成本高且未充分释放大模型中DP的潜力。本文研究在差分隐私约束下的参数高效微调（PEFT）方法，发现其性能接近标准微调，参数量更少，且显著降低隐私泄露。通过引入故意误标的数据投毒实验，直接评估模型记忆行为，结果表明PEFT不仅是一种可行替代方案，更是实现隐私保护、资源高效的大型语言模型微调的有力补充。
原文摘要 · Abstract (English)
Fine-tuning large language models (LLMs) for specific tasks introduces privacy risks, as models may inadvertently memorise and leak sensitive training data. While Differential Privacy (DP) offers a solution to mitigate these risks, it introduces significant computational and performance trade-offs, particularly with standard fine-tuning approaches. Previous work has primarily focused on full-parameter updates, which are computationally intensive and may not fully leverage DPs potential in large models. In this work, we address these shortcomings by investigating Parameter-Efficient Fine-Tuning (PEFT) methods under DP constraints. We show that PEFT methods achieve comparable performance to standard fine-tuning while requiring fewer parameters and significantly reducing privacy leakage. Furthermore, we incorporate a data poisoning experiment involving intentional mislabelling to assess model memorisation and directly measure privacy risks. Our findings indicate that PEFT methods not only provide a promising alternative but also serve as a complementary approach for privacy-preserving, resource-efficient fine-tuning of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。