用教师模型蒸馏学生模型，能有效降低数据记忆风险。
From Teacher to Student: Tracking Memorization Through Model Distillation
- 用大模型蒸馏小模型，减少对训练数据的直接记忆。
- 相比传统微调，蒸馏后模型记忆风险显著降低。
- 适合关注隐私保护的模型部署场景。
大型语言模型（LLMs）会记忆部分训练数据，引发隐私与安全问题。尽管已有研究关注预训练模型的记忆现象，但关于知识蒸馏（KD）如何影响微调任务数据记忆的研究仍较少。本文探讨不同蒸馏方法在将大教师模型微调后，蒸馏到较小学生模型时对任务数据记忆的影响。结果表明，将微调后的大型教师模型蒸馏为较小的学生模型，不仅降低计算成本和模型规模，还能显著减少记忆风险，优于传统的微调方法。
原文摘要 · Abstract (English)
Large language models (LLMs) are known to memorize parts of their training data, raising important concerns around privacy and security. While previous research has focused on studying memorization in pre-trained models, much less is known about how knowledge distillation (KD) affects memorization.In this study, we explore how different KD methods influence the memorization of fine-tuned task data when a large teacher model is distilled into smaller student variants.This study demonstrates that distilling a larger teacher model, fine-tuned on a dataset, into a smaller variant not only lowers computational costs and model size but also significantly reduces the memorization risks compared to standard fine-tuning approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。