arXiv:2607.25063cs.AI2026-07

模型最后预训练数据决定其对后续对齐的响应,影响安全防护能力。

Similar Models Learn Differently: Final-Window Pretraining Shapes Post-Training Beyond SFT

论文配图:Similar Models Learn Differently: Final-Window Pretraining Shapes Post-Training Beyond SFT
图 1 · 摘自论文原文
  • 通过控制最后预训练窗口数据,研究其对模型后续对齐的影响。
  • 相同SFT后表现相近,但后续训练中安全文本分支拒绝有害请求能力显著更强。
  • 最后训练的数据类型决定模型对齐响应,适合关注模型可解释性与安全性的研究者。

开发者通常根据模型在指令跟随、拒绝能力与能力指标上的表现来评估检查点。经监督微调(SFT)后,若两个检查点在相关基准上表现相近,便被视为可互换,均适合进入下一步对齐阶段(如偏好优化)。我们探究这一判断是否忽略了预训练的印记:一种后SFT基准无法揭示、却决定模型如何响应进一步训练的差异。为此,我们在一个部分预训练检查点上开展受控实验,仅改变其最后预训练窗口——5亿令牌(占前序数据0.1%至1%),每支路使用单一数据源:通用网络文本、过滤网络文本、规范性论述、安全文本、数学文本或合成教育文本。所有支路的SFT及后续训练完全一致。尽管SFT后各支路在指令遵循、拒绝能力和能力上差异不足1分,但在后续直接偏好优化和可验证奖励的强化学习更新中,它们走向了截然不同的终点。我们以对有害请求的拒绝率衡量差异:初始时,安全文本支路与通用文本支路拒绝率相近,但训练结束时,前者保留了远多于后者的拒绝能力;其余四支路几乎未获得保护,该效应具有选择性,仅当安全文本处于最后预训练阶段时才出现。该现象在另一模型族中重现。因此,模型最后预训练内容决定了其对齐响应方式,检查点不应仅依据后SFT表现评估,其最后训练数据应一并报告。

原文摘要 · Abstract (English)

Developers judge a model checkpoint by how it behaves. After supervised fine-tuning (SFT), two checkpoints that perform about the same across relevant benchmarks are treated as interchangeable, equally ready for the next alignment stage, typically preference optimization. We ask whether this judgment misses a pretraining imprint: a difference that no post-SFT benchmark reveals, yet that decides how each checkpoint responds to further training. To find out, we run a controlled experiment on the final window of pretraining, the last data trained on before instruction tuning. Six branches fork from one partially pretrained checkpoint and differ only in this window: 500 million tokens, 0.1% to 1% of the tokens that precede it. Each branch trains its window on a single data source: generic web text, filtered web text, normative discourse, safety text, mathematical text, or synthetic educational text. SFT and post-training are then identical. After SFT the branches behave near-identically, within about one point on instruction following, refusal, and capability, yet the same post-training carries them to very different endpoints, under both a direct preference optimization update and a reinforcement learning update with a verifiable reward. We measure this deviation through refusal of harmful requests: when post-training begins the safety text branch refuses no more than the web text branch, yet by the end it has lost far less of its refusal. The other four branches gain little or no protection, so the effect is selective to what the window contained. The protection requires the safety text to arrive last rather than earlier in pretraining, and it reproduces on a second model family. What a model is pretrained on last shapes how it reacts to alignment. Therefore, a checkpoint should not be evaluated by its post-SFT behavior alone, and what it was trained on last should be reported with it.

模型对齐预训练安全防护可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。