mid-training比强化学习更有效,能显著提升大模型的推理能力。
PRISM: Demystifying Retention and Interaction in Mid-Training
- 通过270亿高质量数据进行mid-training,重构90%以上参数以增强推理。
- 数学、代码、科学任务得分提升15-40、5-12、6-13点,推理总分提升3-4倍。
- 适合想提升模型推理能力的研究者,尤其关注训练流程设计的工程团队。
我们提出PRISM,一项对大语言模型中段训练设计选择的全面实证研究。在涵盖四种模型家族(Granite, LLaMA, Mistral, Nemotron-H)、两种架构类型(dense Transformer 和 attention-Mamba 混合)以及3B至24B参数规模的七种基础模型上,通过控制实验发现:约270亿高质量令牌的中段训练可带来数学+15至+40分、代码+5至+12分、科学+6至+13分的稳定提升,同时保持通用性能。完整PRISM到RL的流程使六个推理基准的宏平均分从不足12提升至29-42(3-4倍增长),而直接对基础模型应用强化学习效果差,AIME得分接近零。中段训练阶段的数据构成至关重要:引入科学数据可使后续强化学习在GPQA-Diamond上获得+17至+28分提升,而改变强化学习阶段的数据组合影响不足2分。机制上,中段训练重构超过90%的模型权重,而强化学习仅对约5%参数做稀疏、前置调整。表示分析(CKA)显示强化学习始终保留中段训练的表征几何结构(>0.998 CKA)。关键的是,强化学习施加的权重变化与初始状态无关,但仅在中段训练后的模型上有效,表明中段训练将模型置于一个可被强化学习有效优化的配置中。结果表明,考虑模型保留能力的中段训练对可靠推理提升极为有效,并为构建稳健的中段训练流程提供实践指导。
原文摘要 · Abstract (English)
We present PRISM, a comprehensive empirical study of mid-training design choices for large language models. Through controlled experiments across seven base models spanning four families (Granite, LLaMA, Mistral, Nemotron-H), two architecture types (dense Transformer and attention-Mamba hybrid), and scales from 3B to 24B parameters, we show that mid-training on approximately 27B high-quality tokens yields consistent gains of +15 to +40 points on math, +5 to +12 points on code, and +6 to +13 points on science benchmarks while preserving general performance. The full PRISM to RL pipeline improves macro-average across six reasoning benchmarks from under 12 to 29-42 (a 3-4x improvement), whereas RL applied directly to most of the base models remains substantially less effective, with AIME scores near zero. Data composition matters most at mid-training, not RL: including science data during mid-training unlocks +17 to +28 point GPQA-Diamond gains during RL, while changing the RL mix produces less than 2 point differences. Mechanistically, mid-training densely restructures over 90% of model weights, while RL makes sparse, front-loaded refinements to approximately 5% of parameters. Representation analysis (CKA) confirms that RL consistently preserves mid-training's representational geometry (over 0.998 CKA) across architectures. Crucially, RL applies identical weight changes regardless of starting point, yet only succeeds on mid-trained models, consistent with mid-training placing the model in a configuration from which RL can effectively improve performance. Our results demonstrate that retention-aware mid-training is highly effective for reliable reasoning enhancement and provide practical guidance for designing robust mid-training pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。