LoRA微调可显著降低大模型记忆风险，同时保持良好性能。
Leaner Training, Lower Leakage: Revisiting Memorization in LLM Fine-Tuning with LoRA
- 采用松弛相似度度量，对比不同微调策略的记忆行为。
- LoRA微调下记忆风险远低于全量微调，且不随模型规模或数据重复变化。
- 适合关注隐私保护与高效微调的AI研发人员参考。
大语言模型中的记忆现象使其易受数据提取攻击。尽管预训练阶段的记忆已获广泛研究，但对微调阶段尤其是参数高效方法LoRA微调的影响研究仍较少。本文重新审视微调中的记忆问题，发现不同微调策略下记忆行为存在意外差异：模型规模和数据重复性等在预训练及全量微调中影响显著的因素，在LoRA微调中不再呈现相同趋势。通过引入更宽松的基于相似度的记忆度量，我们证明在保持优异任务性能的同时，LoRA显著降低了记忆风险。
原文摘要 · Abstract (English)
Memorization in large language models (LLMs) makes them vulnerable to data extraction attacks. While pre-training memorization has been extensively studied, fewer works have explored its impact in fine-tuning, particularly for LoRA fine-tuning, a widely adopted parameter-efficient method. In this work, we re-examine memorization in fine-tuning and uncover a surprising divergence from prior findings across different fine-tuning strategies. Factors such as model scale and data duplication, which strongly influence memorization in pre-training and full fine-tuning, do not follow the same trend in LoRA fine-tuning. Using a more relaxed similarity-based memorization metric, we demonstrate that LoRA significantly reduces memorization risks compared to full fine-tuning, while still maintaining strong task performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。