解析大模型预训练与微调的数据质量机制，揭示不同阶段数据规模的最优策略。
Theoretical Perspectives on Data Quality and Synergistic Effects in Pre- and Post-Training Reasoning Models
- 通过上下文权重预测任务理论分析变压器模型，揭示预训练数据平衡性对潜在能力的激活作用。
- 小规模高质量微调数据更优，过大数据集反而稀释预训练信号，而强化学习依赖大规模数据。
- 研究适合理解大模型训练机制的研究者，尤其关注数据质量与训练阶段匹配问题。
大型语言模型（LLMs）在海量数据上进行预训练，随后通过监督微调（SFT）或强化学习（RL）进行后训练。最佳实践强调预训练数据的规模与多样性，但后训练则不同：SFT依赖较小且高质量的数据集，而RL更受益于数据规模，大量反馈常胜过标签质量。然而，为何预训练和RL需要大规模数据，为何SFT在小数据上表现更好，以及何为高质量的SFT数据仍不明确。本文针对线性回归的上下文权重预测任务，理论上分析了变压器模型。结果表明：(i) 平衡的预训练数据可诱导出后期在后训练中被激活的潜在能力；(ii) SFT在少量对预训练模型具有挑战性的样本上效果最佳，过大的SFT数据集可能稀释有价值的预训练信号。相比之下，RL在大规模但不过于困难的数据上最有效。我们在大型非线性变压器架构上验证了这些理论发现。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are pretrained on massive datasets and later instruction-tuned via supervised fine-tuning (SFT) or reinforcement learning (RL). Best practices emphasize large, diverse pretraining data, whereas post-training operates differently: SFT relies on smaller, high-quality datasets, while RL benefits more from scale, with larger amounts of feedback often outweighing label quality. Yet it remains unclear why pretraining and RL require large datasets, why SFT excels on smaller ones, and what defines high-quality SFT data. In this work, we theoretically analyze transformers trained on an in-context weight prediction task for linear regression. Our analysis reveals several key findings: $(i)$ balanced pretraining data can induce latent capabilities later activated during post-training, and $(ii)$ SFT learns best from a small set of examples challenging for the pretrained model, while excessively large SFT datasets may dilute informative pretraining signals. In contrast, RL is most effective on large-scale data that is not overly difficult for the pretrained model. We validate these theoretical insights with experiments on large nonlinear transformer architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。