提出新方法量化大模型训练后遗忘与知识回传,揭示不同训练阶段的影响。
Mapping Post-Training Forgetting in Language Models at Scale
- 按样本追踪正确变错误(1→0)和错误变正确(0→1)来度量遗忘与回传
- 指令微调后数学逻辑能力显著提升,但小数据下效果有限,大模型敏感性高
- 模型合并无法稳定缓解遗忘,为训练策略优化提供可测量依据
大规模训练后微调正推动语言模型能力的显著提升,但其对预训练知识的影响尚不清晰。并非所有遗忘都等价:记住一个事实(如美国总统或API调用)并不抵消遗忘另一个事实。为此,我们提出一种基于样本的分析范式,通过统计1→0转变(训练前正确、训练后错误)衡量遗忘,0→1转变(训练前错误、训练后正确)衡量反向迁移。传统任务平均值会掩盖这些变化。针对多项选择题基准,我们引入机会调整版本,从前后准确率中减去随机猜测的预期贡献。该框架应用于多个微调阶段、模型规模与数据量。大规模分析表明:(1) 域持续预训练导致中等程度遗忘,伴随低至中等的反向迁移;(2) 对基础模型及指令微调模型应用强化学习/监督微调(RL/SFT),在数学与逻辑任务上实现中到大幅反向迁移,整体遗忘水平为低至中等;(3) 将RL/SFT应用于指令微调模型时,对数据规模敏感:小数据下遗忘与反向迁移均较小;大数据下影响混杂,需更严谨控制进一步研究;(4) 模型合并无法可靠减轻遗忘。整体而言,本框架为规模化评估训练后知识演变提供了实用标尺,助力构建通用智能系统。
原文摘要 · Abstract (English)
Scaled post-training now drives many of the largest capability gains in language models (LMs), yet its effect on pretrained knowledge remains poorly understood. Not all forgetting is equal: Forgetting one fact (e.g., a U.S. president or an API call) does not "average out" by recalling another. Hence, we propose a sample-wise paradigm to measure what is forgotten and when backward transfer occurs. Our metric counts 1->0 transitions (correct before post-training, incorrect after) to quantify forgetting and 0->1 transitions to quantify backward transfer. Traditional task averages conflate these effects and obscure large changes. For multiple-choice benchmarks, we add chance-adjusted variants that subtract the expected contribution of random guessing from pre- and post-training accuracies. We apply this framework across post-training stages, model sizes, and data scales. Our large-scale analysis shows that: (1) Domain-continual pretraining induces moderate forgetting with low-to-moderate backward transfer; (2) RL/SFT post-training applied to base models and Instruction tuning yields moderate-to-large backward transfer on math and logic with overall low-to-moderate forgetting; (3) Applying RL/SFT to instruction-tuned models is sensitive on data scale: at small scales, both forgetting and backward transfer are small; at larger scales, effects are mixed and warrant further study with better controls; (4) Model merging does not reliably mitigate forgetting. Overall, our framework offers a practical yardstick for mapping how post-training alters pretrained knowledge at scale -- enabling progress towards generally capable AI systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。