LLM训练中下游性能波动大，通过集成相邻检查点可稳定效果。
Instability in Downstream Task Performance During LLM Pretraining
- 用相邻检查点平均或集成缓解下游任务性能波动
- 集成方法使性能稳定性提升，且不改变训练过程
- 适合关注模型可靠性与部署稳定性的研究者
训练大型语言模型时，通常在训练过程中监控下游任务表现，并选择验证分数最高的检查点。然而，下游指标常出现显著波动，难以确定最佳性能模型。本文针对在多样网页规模语料上训练的LLM，实证分析了下游任务性能的稳定性，发现任务得分在整体和样本层面均频繁波动。为解决此问题，我们研究了两种事后检查点集成方法：检查点平均和集成，基于邻近检查点聚合可降低性能波动的假设。我们从实证和理论上证明，这些方法能在不修改训练流程的前提下，有效提升下游性能稳定性。
原文摘要 · Abstract (English)
When training large language models (LLMs), it is common practice to track downstream task performance throughout the training process and select the checkpoint with the highest validation score. However, downstream metrics often exhibit substantial fluctuations, making it difficult to identify the checkpoint that truly represents the best-performing model. In this study, we empirically analyze the stability of downstream task performance in an LLM trained on diverse web-scale corpora. We find that task scores frequently fluctuate throughout training, both at the aggregate and example levels. To address this instability, we investigate two post-hoc checkpoint integration methods: checkpoint averaging and ensemble, motivated by the hypothesis that aggregating neighboring checkpoints can reduce performance volatility. We demonstrate both empirically and theoretically that these methods improve downstream performance stability without requiring any changes to the training procedure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。