预训练模型选得分最高的未必最好，好模型更耐扰动。
Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack
- 用解空间密度衡量模型鲁棒性，优于传统损失值
- 300亿参数专家混合模型中，高密度检查点下游表现更优
- 适合关注模型稳定性与微调效果的研究者
语言模型检查点通常按预训练损失或基准分数选择，假设最高分检查点在后续训练中仍为最佳起点。我们在一个完整的300亿参数专家混合（MoE）训练流水线中发现，这一假设可能失效。经过完整下游训练后表现更好的检查点，其解空间密度更高，即在局部权重扰动下仍能保持下游性能。这表明，解空间密度是比单纯损失值更可靠的检查点筛选指标。
原文摘要 · Abstract (English)
Language-model checkpoints are commonly selected by pretraining loss or benchmark scores, assuming that the highest-scoring checkpoint will remain the best starting point for subsequent training. We show that this assumption can fail in a full 30B mixture-of-experts training pipeline. The checkpoints that perform better after the full downstream training stack also have higher solution density, i.e., retain downstream performance under local weight perturbations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。