用无上下文生成数据缓解微调时的遗忘问题
Context-Free Synthetic Data Mitigates Forgetting
- 通过无上下文生成数据近似估计模型参数变化的KL散度
- 在OLMo-1B和R1-Distill-Llama-8B上显著减少性能退化
- 适合资源受限下保护预训练模型能力的研究者
微调语言模型常导致其在其他任务上的性能下降，即（灾难性）遗忘。我们关注仅能访问模型权重而无训练数据/配方的场景。一种自然方法是惩罚原模型与新模型之间的KL散度。我们的核心发现是：一种称为无上下文生成的过程，可实现该KL散度的近似无偏估计。实验表明，在微调数据中加入无上下文生成的数据，能有效缓解遗忘现象，涵盖两种设置：(a) 保持仅预训练模型的零样本性能；(b) 保持思维型模型的推理性能。相比之下，上下文合成数据甚至部分预训练数据效果较差。我们还考察了生成温度、数据比例等影响因素。结果基于 OLMo-1B（预训练模型设置）与 R1-Distill-Llama-8B（推理模型设置）呈现。
原文摘要 · Abstract (English)
Fine-tuning a language model often results in a degradation of its existing performance on other tasks, due to a shift in the model parameters; this phenomenon is often referred to as (catastrophic) forgetting. We are interested in mitigating this, in settings where we only have access to the model weights but no access to its training data/recipe. A natural approach is to penalize the KL divergence between the original model and the new one. Our main realization is that a simple process - which we term context-free generation - allows for an approximate unbiased estimation of this KL divergence. We show that augmenting a fine-tuning dataset with context-free generations mitigates forgetting, in two settings: (a) preserving the zero-shot performance of pretrained-only models, and (b) preserving the reasoning performance of thinking models. We show that contextual synthetic data, and even a portion of the pretraining data, are less effective. We also investigate the effect of choices like generation temperature, data ratios etc. We present our results for OLMo-1B for pretrained-only setting and R1-Distill-Llama-8B for the reasoning setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。