arXiv:2510.15990cs.LGcs.AI2025-10被引 5

GRPO只能强化预训练偏差,无法突破模型能力边界。

Can GRPO Help LLMs Transcend Their Pretraining Origin?

  • GRPO本质是保守重加权,受限于预训练分布
  • 仅当任务与预训练偏好一致时才能提升推理表现
  • 适合优化已知领域,不适用于全新任务

基于可验证奖励的强化学习(RLVR)主要由组相对策略优化(GRPO)驱动,是提升大语言模型(LLM)推理能力的主流方法。尽管广泛应用,其效果常不稳定:模型在数学等某领域显著提升,但在医学等另一领域却无进展。这引发关键问题:在何种条件下GRPO能提升推理并实现分布外(OOD)泛化?本文从数据分布视角展开研究。理论证明GRPO是一种保守重加权机制,受基础模型分布限制,无法发现完全新颖解法。通过精心设计的控制实验,我们训练从头开始的Transformer,评估其在推理深度、输入长度、标记表示和组合性上的泛化能力。结果表明:仅有当目标任务与预训练偏见对齐时,才出现分布外改进;而在分布内(ID)任务上,性能饱和后提升逐渐消失。该发现将GRPO重新定义为增强预训练偏见的工具,而非通用推理增强器。这为开发突破预训练局限的新算法提供了方向。

原文摘要 · Abstract (English)

Reinforcement Learning with Verifiable Rewards (RLVR), primarily driven by the Group Relative Policy Optimization (GRPO) algorithm, is a leading approach for enhancing the reasoning abilities of Large Language Models (LLMs). Despite its wide adoption, GRPO's gains are often inconsistent; for instance, a model may show significant improvement in one reasoning domain, like mathematics, yet remain stagnant in another, such as medicine. This inconsistency raises a critical question: under what conditions does GRPO improve reasoning and generalize out-of-distribution (OOD)? We investigate this from a data distribution perspective. We first prove theoretically that GRPO is a conservative reweighting scheme, bounded by the base model's distribution and thus unable to discover completely novel solutions. We further validate this in carefully designed controlled studies by training transformers from scratch, evaluating generalization across reasoning depth, input length, token representation, and compositionality. Our results provide a principled explanation for GRPO's boundaries: OOD improvement emerges only when the target task aligns with the model's pretrained biases, while gains on in-distribution (ID) tasks diminish as performance saturates. This reframes GRPO not as a universal reasoning enhancer but as a tool that sharpens pretraining biases. Our findings motivate future development of algorithms that can expand a model's capabilities beyond its pretraining origin.

强化学习大模型推理能力泛化性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。