大模型后训练本质是数据分布拟合,而非真正智能提升。
Post-training is (Massive) Supervised Learning
- 将预训练模型从零开始后训练,仍可达到显著性能
- 随机初始化模型经后训练后在数学与代码任务上表现不俗
- 提示未来应关注模型自学习能力,而非固定行为调优
当前大模型训练主流依赖大规模后训练阶段,包括监督微调(SFT)与强化学习(RL)。本文认为,该方法实质上回归了BERT时代的“预训练+微调”范式,即通过显式适配目标行为与评测基准来优化模型。我们回顾了大模型的发展历程,指出当前状态与早期大模型高度相似——性能严重依赖于对分布内数据的拟合。为实证此观点,我们在现代推理数据集上对预训练模型与随机初始化模型进行微调,并在数学和代码竞赛基准上评估其表现。结果表明,从零开始后训练的模型也能取得非平凡的性能。研究暗示,现有后训练方法主要起分布拟合作用。最后,我们主张:要发展通用能力强的模型与系统,需超越针对特定行为的大量后训练,转而探索让模型‘学会如何学习’的训练机制。
原文摘要 · Abstract (English)
The prevailing paradigm for training LLMs has evolved to rely on a massive post-training phase consisting of SFT and RL. In this position paper, we argue that this methodology effectively marks a reversion to the ``pre-train then fine-tune'' approach of the BERT era, explicitly tailoring models to the desired behaviors and specific benchmarks on which they are evaluated. We begin with a historical overview of LLMs, describing the different phases of the LLM evolution. We argue that the current landscape is remarkably similar to the early days of LLMs, where task performance heavily relied on fitting the models to in-distribution datasets. To empirically demonstrate this, we compare pre-trained models to randomly initialized ones, by fine-tuning both variants on modern reasoning datasets and evaluating them on competitive math and code benchmarks. We show that models post-trained from scratch yield highly non-trivial performance. Our findings suggest that current post-training methodologies function primarily as a distribution-fitting mechanism. We finish by positing that developing generally capable models and systems requires moving beyond extensive post-training for predefined behaviors, shifting instead toward training procedures where models ``learn how to learn''.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。