用平滑优化减少模型遗忘，提升微调和量化后性能
Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting
- 通过平滑感知优化引导训练走向更平坦的极小值
- 在5个数据集上实现最多80%的遗忘减少
- 适合关注模型稳定性与低精度部署的研究者
预训练优化器通常追求最强的基础模型，但忽略了模型几何结构对后续参数更新中能力保留的影响。本文研究三种导向平坦极小值的预训练优化方法：尖锐感知最小化（SAM）、大学习率和缩短学习率退火周期。在20M至150M参数的模型上，这些方法在五个常见数据集上的微调后表现显著提升，遗忘程度最多降低80%。在更大规模下同样有效：对已有的OLMo-2-1B检查点进行短时SAM训练，可使MetaMath微调后遗忘减少31%，4比特量化后减少40%。
原文摘要 · Abstract (English)
Pretraining optimizers are tuned to produce the strongest possible base model, on the assumption that a stronger starting point yields a stronger model after subsequent changes like post-training and quantization. This overlooks the geometry of the base model which controls how much of the base model's capabilities survive subsequent parameter updates. We study three pretraining optimization approaches that bias optimization toward flatter minima: Sharpness-Aware Minimization (SAM), large learning rates, and shortened learning rate annealing periods. Across model sizes ranging from 20M to 150M parameters, we find that these interventions consistently improve downstream performance after post-training on five common datasets with up to 80% less forgetting. These principles hold at scale: a short SAM mid-training phase applied to an existing OLMo-2-1B checkpoint reduces forgetting by 31% after MetaMath post-training and by 40% after 4-bit quantization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。