微调时注入1%预训练数据可防止模型遗忘通用知识
Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection
- 在微调数据中加入少量预训练数据以缓解遗忘
- 仅需1%预训练数据即可完全避免知识遗忘
- 适用于资源有限但需保持通用能力的微调场景
一种常见的方法是用目标领域数据对预训练语言模型进行微调，使其在该领域表现良好。然而微调面临两大挑战：（i）当目标数据量有限时，模型会迅速过拟合；（ii）模型会偏离原始状态，遗忘预训练数据及其中蕴含的通用知识。本文旨在为不同目标领域、可用目标数据量和模型规模下，量化过拟合与遗忘现象的规律。通过测量将预训练数据注入微调数据混合中的效率，发现仅需在微调数据中加入1%的预训练数据，即可有效防止模型遗忘预训练集。
原文摘要 · Abstract (English)
A widespread strategy to obtain a language model that performs well on a target domain is to finetune a pretrained model to perform unsupervised next-token prediction on data from that target domain. Finetuning presents two challenges: (i) if the amount of target data is limited, as in most practical applications, the model will quickly overfit, and (ii) the model will drift away from the original model, forgetting the pretraining data and the generic knowledge that comes with it. We aim to derive scaling laws that quantify these two phenomena for various target domains, amounts of available target data, and model scales. We measure the efficiency of injecting pretraining data into the finetuning data mixture to avoid forgetting and mitigate overfitting. A key practical takeaway from our study is that injecting as little as 1% of pretraining data in the finetuning data mixture prevents the model from forgetting the pretraining set.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。