提出MaP框架，让大模型预训练评估更稳定可靠。
MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamics
- 用检查点合并平滑参数空间，降低训练随机性影响。
- 引入Pass@k指标，减少评估噪声，提升结果稳定性。
- 适合关注模型训练过程分析的研究者和开发者。
可靠的评估是大语言模型（LLMs）发展的基础，但预训练过程中的评估存在显著不稳定性，掩盖了真实的训练动态。本文系统诊断该问题，发现不稳定性来自两个源头：训练随机性引发的参数波动，以及评估协议噪声导致的测量偏差。为此，我们提出双管齐下的MaP框架，融合检查点合并（Checkpoint Merging）与Pass@k指标。检查点合并通过对近期模型权重求平均，平滑参数空间；Pass@k则提供低方差、稳健的模型能力统计估计。大量实验表明，MaP能显著改善性能曲线平滑度，降低不同运行间的方差，确保模型排名更一致。最终，MaP为观测大模型训练动态提供了更可靠、更忠实的视角，为LLM研究奠定关键的实证基础。
原文摘要 · Abstract (English)
Reliable evaluation is fundamental to the progress of Large Language Models (LLMs), yet the evaluation process during pre-training is plagued by significant instability that obscures true learning dynamics. In this work, we systematically diagnose this instability, attributing it to two distinct sources: \textit{Parameter Instability} from training stochasticity and \textit{Evaluation Instability} from noisy measurement protocols. To counteract both sources of noise, we introduce \textbf{MaP}, a dual-pronged framework that synergistically integrates checkpoint \underline{M}erging \underline{a}nd the \underline{P}ass@k metric. Checkpoint merging smooths the parameter space by averaging recent model weights, while Pass@k provides a robust, low-variance statistical estimate of model capability. Extensive experiments show that MaP yields significantly smoother performance curves, reduces inter-run variance, and ensures more consistent model rankings. Ultimately, MaP provides a more reliable and faithful lens for observing LLM training dynamics, laying a crucial empirical foundation for LLM research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。