提前引入强化学习能显著提升大模型训练效果，且优于传统流程。
RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training
- 在预训练中途直接应用强化学习，效果不亚于完整训练流程。
- 针对难题优化数据组合比增大模型规模更有效提升强化学习表现。
- 强化学习可扩展模型能力分布，而微调反而会降低通用能力。
标准的大语言模型训练流程仅在预训练和监督微调（SFT）后才引入强化学习（RL）。本文挑战这一惯例，从零开始训练模型，并将RL、SFT以及先SFT后RL的策略直接应用于中间预训练检查点。实验表明，强化学习在早期阶段即有效，且常在早期便达到完整SFT→RL流程的性能水平。在更难任务上，有针对性地设计预训练数据组合是提升强化学习效果的关键因素，甚至超过模型规模的影响。除推理准确率外，直接对基础检查点施加强化学习可扩展模型的能力分布；而近期报道的“锐化效应”仅在强化学习位于微调之后时出现。强化学习本身基本不影响模型的通用能力，但微调会导致其下降。最后，通过并行平均融合强化学习与微调目标，该方法在所有指标上均优于其他训练方式，同时保持了模型的通用能力。这些结果提示，大语言模型训练或应更广泛地使用强化学习。
原文摘要 · Abstract (English)
The standard LLM training pipeline applies reinforcement learning (RL) only after pre-training and supervised fine-tuning (SFT). We question this status quo by training a LLM from scratch and applying RL, SFT, and SFT followed by RL directly to intermediate pre-training checkpoints. We find that RL is effective very early, and often matches the full SFT$\to$RL pipeline early as well. Through experiments on harder problems, we find that targeted pre-training data composition is a strong lever for RL effectiveness, even more so than model scale. Beyond reasoning accuracy, applying RL directly to base checkpoints expands the model's distribution; the sharpening effect reported in recent work arises only when RL follows SFT. The general capabilities of the model remain essentially unchanged by RL, while they degrade following SFT. Finally, we merge RL and SFT objectives by parallel averaging, which outperforms across all other training methods discussed, across metrics, while preserving general capabilities. Together, these results suggest that LLM training might benefit from an expanded use of RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。