通过分层提示引导,让大模型突破推理能力瓶颈
Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning
- 引入分层提示机制,在模型停滞时逐步提供抽象到具体的指导
- 在AIME24上使Qwen2.5-Math-7B的解题准确率提升44.3%
- 适合需要增强复杂推理能力的大模型研究与应用
基于可验证奖励的强化学习已成为提升大语言模型(LLM)复杂推理能力的有效方法。然而,这类方法受限于“学习悬崖”现象:当问题远超模型当前能力时,模型持续失败,导致零奖励信号。在GRPO等策略优化算法中,这会使优势计算归零,使难题无法被学习梯度捕捉,阻碍进步。为此,我们提出Scaf-GRPO(分层组相对策略优化),一种渐进式训练框架,仅在模型独立学习陷入停滞时提供最小引导。该框架首先诊断学习停滞,随后注入由抽象概念到具体步骤的分层提示,帮助模型自主构建有效解。在多个挑战性数学基准上的实验表明,Scaf-GRPO显著提升性能,使Qwen2.5-Math-7B在AIME24上的pass@1分数相比基线GRPO提升44.3%。该结果证明,本框架为解锁模型解决此前无法应对的问题提供了稳健有效的路径,是推动大模型自主推理能力边界的关键一步。
原文摘要 · Abstract (English)
Reinforcement learning from verifiable rewards has emerged as a powerful technique for enhancing the complex reasoning abilities of Large Language Models (LLMs). However, these methods are fundamentally constrained by the ''learning cliff'' phenomenon: when faced with problems far beyond their current capabilities, models consistently fail, yielding a persistent zero-reward signal. In policy optimization algorithms like GRPO, this collapses the advantage calculation to zero, rendering these difficult problems invisible to the learning gradient and stalling progress. To overcome this, we introduce Scaf-GRPO (Scaffolded Group Relative Policy Optimization), a progressive training framework that strategically provides minimal guidance only when a model's independent learning has plateaued. The framework first diagnoses learning stagnation and then intervenes by injecting tiered in-prompt hints, ranging from abstract concepts to concrete steps, enabling the model to construct a valid solution by itself. Extensive experiments on challenging mathematics benchmarks demonstrate Scaf-GRPO's effectiveness, boosting the pass@1 score of the Qwen2.5-Math-7B model on the AIME24 benchmark by a relative 44.3% over a vanilla GRPO baseline. This result demonstrates our framework provides a robust and effective methodology for unlocking a model's ability to solve problems previously beyond its reach, a critical step towards extending the frontier of autonomous reasoning in LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。