通过预训练空间强化学习，提升大模型的推理能力。
From $P(y|x)$ to $P(y)$: Investigating Reinforcement Learning in Pre-train Space
- 在预训练空间直接优化输出分布P(y)，突破原有推理瓶颈。
- 负样本强化使错误推理空间减少，反思思考提升14.89倍。
- 适合希望提升大模型推理能力的研究者与工程师。
尽管基于可验证奖励的强化学习（RLVR）通过优化条件分布P(y|x)显著增强大模型的推理能力，但其潜力受限于基础模型原有的输出分布。通过在预训练空间中优化边缘分布P(y)，可编码推理能力并保留广泛探索能力。然而，传统预训练依赖静态语料进行被动学习，导致分布偏移，阻碍针对性推理提升。本文提出PreRL（预训练空间强化学习），直接对P(y)实施奖励驱动的在线更新。理论与实证验证了log P(y)与log P(y|x)之间强梯度对齐，证明PreRL是标准RL的有效替代。进一步发现：预训练空间中的负样本强化（NSR）是推理增强的关键机制。NSR-PreRL能快速剪枝错误推理路径，激发内在反思行为，使转换与反思思考分别提升14.89倍和6.54倍。基于此，提出双空间强化学习（DSRL），采用策略重生策略，先以NSR-PreRL初始化模型以扩展推理范围，再转入标准RL进行精细化优化。大量实验表明，DSRL持续优于强基线，证明预训练空间剪枝能有效引导策略进入更优的正确推理子空间。
原文摘要 · Abstract (English)
While reinforcement learning with verifiable rewards (RLVR) significantly enhances LLM reasoning by optimizing the conditional distribution P(y|x), its potential is fundamentally bounded by the base model's existing output distribution. Optimizing the marginal distribution P(y) in the Pre-train Space addresses this bottleneck by encoding reasoning ability and preserving broad exploration capacity. Yet, conventional pre-training relies on static corpora for passive learning, leading to a distribution shift that hinders targeted reasoning enhancement. In this paper, we introduce PreRL (Pre-train Space RL), which applies reward-driven online updates directly to P(y). We theoretically and empirically validate the strong gradient alignment between log P(y) and log P(y|x), establishing PreRL as a viable surrogate for standard RL. Furthermore, we uncover a critical mechanism: Negative Sample Reinforcement (NSR) within PreRL serves as an exceptionally effective driver for reasoning. NSR-PreRL rapidly prunes incorrect reasoning spaces while stimulating endogenous reflective behaviors, increasing transition and reflection thoughts by 14.89x and 6.54x, respectively. Leveraging these insights, we propose Dual Space RL (DSRL), a Policy Reincarnation strategy that initializes models with NSR-PreRL to expand the reasoning horizon before transitioning to standard RL for fine-grained optimization. Extensive experiments demonstrate that DSRL consistently outperforms strong baselines, proving that pre-train space pruning effectively steers the policy toward a refined correct reasoning subspace.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。