构建可扩展的强化学习数据流水线，将海量文本转化为高质量问答对用于模型训练。
Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels
- 设计自动化流水线，将大规模预训练文档转为多样且可验证的问答数据
- 生成120万条跨9个领域的问答数据，显著提升模型性能
- 相较持续预训练，用百倍更少的文本达到同等效果，适合高效训练研究者
大型语言模型通过在海量文本语料上进行模仿学习取得了显著成功，但这种范式导致了训练与生成之间的差距，并限制了模型的鲁棒推理能力。强化学习（RL）提供了一种更高效的数据利用方案，有望弥合这一差距，但其应用受限于关键的数据瓶颈：现有RL数据集规模和多样性远低于网络级预训练语料。为此，我们提出Webscale-RL流水线，一种可扩展的数据引擎，可系统性地将大规模预训练文档转化为数百万条多样化、可验证的问答对，用于强化学习。基于该流水线，我们构建了包含120万例、覆盖9个以上领域的Webscale-RL数据集。实验表明，基于该数据集训练的模型在多个基准测试中显著优于持续预训练和强数据精炼基线。值得注意的是，使用该数据集的强化学习训练效率极高，仅需持续预训练所需1/100的词元即可达到相当性能。本工作为将强化学习扩展至预训练级别提供了可行路径，推动更强大且高效的语言模型发展。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have achieved remarkable success through imitation learning on vast text corpora, but this paradigm creates a training-generation gap and limits robust reasoning. Reinforcement learning (RL) offers a more data-efficient solution capable of bridging this gap, yet its application has been constrained by a critical data bottleneck: existing RL datasets are orders of magnitude smaller and less diverse than web-scale pre-training corpora. To address this, we introduce the Webscale-RL pipeline, a scalable data engine that systematically converts large-scale pre-training documents into millions of diverse, verifiable question-answer pairs for RL. Using this pipeline, we construct the Webscale-RL dataset, containing 1.2 million examples across more than 9 domains. Our experiments show that the model trained on this dataset significantly outperforms continual pretraining and strong data refinement baselines across a suite of benchmarks. Notably, RL training with our dataset proves substantially more efficient, achieving the performance of continual pre-training with up to 100$\times$ fewer tokens. Our work presents a viable path toward scaling RL to pre-training levels, enabling more capable and efficient language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。