让大模型在无标签数据中主动学习推理,突破通用智能预训练瓶颈。
PretrainZero: Reinforcement Active Pretraining
- 基于预训练语料主动识别高信息量内容,用强化学习自主推理补全
- 无需标注或奖励模型,在维基百科上直接预训练3~30B模型,提升推理能力
- 适用于需要通用推理能力的研究者,尤其适合构建基础推理模型
模仿人类主动从通用经验中学习以实现通用人工智能一直是人类梦想。近期基于强化学习(RL)的大模型在软件和数学等任务上展现出专家级能力,但仍严重依赖特定领域的可验证奖励,限制了通用推理能力的拓展。本文提出PretrainZero,一种基于预训练语料的强化主动学习框架,将RL从领域特定的后训练扩展到通用预训练。其特点包括:1)主动预训练:受人类主动学习启发,模型统一学习推理策略,主动识别预训练语料中合理且高信息量的内容,并通过强化学习预测这些内容;2)自监督学习:无需任何可验证标签、预训练奖励模型或监督微调,直接在通用维基百科语料上对3~30B规模的基础模型进行强化预训练,显著突破通用推理的验证数据壁垒;3)验证扩展性:通过逐步攻克更难的掩码片段,大幅提升预训练模型的通用推理能力。在强化预训练中,PretrainZero使Qwen3-4B-Base在MMLU-Pro、SuperGPQA和数学平均基准上分别提升8.43、5.96和10.60分。在后训练阶段,预训练模型还可作为下游强化学习视觉推理任务的推理基础模型。
原文摘要 · Abstract (English)
Mimicking human behavior to actively learning from general experience and achieve artificial general intelligence has always been a human dream. Recent reinforcement learning (RL) based large-thinking models demonstrate impressive expert-level abilities, i.e., software and math, but still rely heavily on verifiable rewards in specific domains, placing a significant bottleneck to extend the performance boundary of general reasoning capabilities. In this work, we propose PretrainZero, a reinforcement active learning framework built on the pretraining corpus to extend RL from domain-specific post-training to general pretraining. PretrainZero features the following characteristics: 1) Active pretraining: inspired by the active learning ability of humans, PretrainZero learns a unified reasoning policy to actively identify reasonable and informative contents from pretraining corpus, and reason to predict these contents by RL. 2) Self-supervised learning: without any verifiable labels, pretrained reward models, or supervised fine-tuning, we directly pretrain reasoners from 3 to 30B base models on the general Wikipedia corpus using RL, significantly breaking the verification data-wall for general reasoning. 3) Verification scaling: by tackling increasingly challenging masked spans, PretrainZero substantially enhances the general reasoning abilities of pretrained base models. In reinforcement pretraining, PretrainZero improves Qwen3-4B-Base for 8.43, 5.96 and 10.60 on MMLU-Pro, SuperGPQA and math average benchmarks. In post-training, the pretrained models can also serve as reasoning foundation models for downstream RLVR tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。