学习率大小影响大模型微调时的灾难性过训练。
(How) Learning Rates Regulate Catastrophic Overtraining
- 通过学习率隐式正则化分析微调中的遗忘机制。
- 大步长和小步长微调收敛到性质不同的模型。
- 学习率衰减会加剧过训练，适合研究优化动态者阅读。
监督微调（SFT）是大语言模型（LLM）后训练的常见第一阶段，用于教会模型遵循指令并塑造其作为助手的行为。然而，长期预训练后的SFT可能损害模型的基础能力，这一现象称为灾难性过训练（Springer et al., 2025）。为理解该现象，我们从学习率的隐式正则化视角研究了微调中的灾难性遗忘。对于达到相同SFT损失的模型，我们发现学习率调控优化路径：使用大步长与小步长微调会收敛到性质不同的模型。接着，我们将遗忘与过训练关联：学习率衰减会增加预训练模型的尖锐度，从而在SFT中加剧灾难性遗忘，导致过训练。我们的研究揭示了大模型过训练的机制，并深化了对预训练与微调优化动态相互作用的理解。
原文摘要 · Abstract (English)
Supervised fine-tuning (SFT) is a common first stage of LLM post-training, teaching the model to follow instructions and shaping its behavior as a helpful assistant. At the same time, SFT may harm the fundamental capabilities of an LLM, particularly after long pretraining: a phenomenon known as catastrophic overtraining (Springer et al., 2025). To understand overtraining, we first investigate catastrophic forgetting in finetuning through the lens of implicit regularization of the learning rate. For models trained to the same SFT loss, we identify how the learning rate mediates optimization: finetuning with large and small steps converges to qualitatively different models. Next, we link forgetting to overtraining: learning rate decay increases the sharpness of the pretrained model, which in turn exacerbates catastrophic forgetting during SFT, leading to overtraining. Our findings paint a picture of the overtraining mechanism in LLMs and broadly contribute to the understanding of the interplay between optimization dynamics during pretraining and finetuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。