针对数学推理强化学习中失败提示梯度消失问题，提出采样时低秩修复机制。
LoRA Scaffolded Policy Optimization (LSPO): A Sampling-Time Low-Rank Scaffold for Recovering Reinforcement-Learning Gradient on Zero-Reward Cliff Prompts
- 在采样阶段检测失败提示，用小规模低秩适配器快速拟合正确解
- 通过重采样成功完成并修正重要性权重，恢复梯度信息
- 适合提升大模型在高难度数学题上的推理能力，尤其适用于零奖励场景
基于可验证奖励的强化学习在数学推理中存在结构性盲区：对于所有采样轨迹均失败的‘悬崖’类提示，组内归一化的优势值恒为零，导致GRPO无法生成梯度，而这正是模型能力边界处的关键提示。本文提出采样时低秩支架优化（LSPO），每轮强化学习中检测悬崖提示，通过简短监督训练拟合其正确解的小型低秩适配器（LoRA），以基模型+适配器重新采样，将成功生成内容拼回强化学习批次并进行重要性校正，最后仅对基模型执行GRPO更新；适配器仅接收监督梯度并在检查点丢弃，最终获得纯基模型。在DeepMath-103K数据集上，使用DeepSeek-R1-Distill-Qwen-1.5B模型，每组对比运行5个种子，在1000步报告周期下，LSPO的5种子平均表现优于或持平于DAPO基线在全部16个（基准，pass@k）评估单元中的表现（15次严格胜出，1次平局），最大提升达+10.7点（AIME24/pass@4）、+6.7点（AIME24 & AIME26/pass@16），以及+2.4点（MATH500/pass@1）；16个单元平均提升+3.8点。
原文摘要 · Abstract (English)
Reinforcement learning from verifiable rewards (RLVR) for mathematical reasoning suffers from a structural blind spot: on "cliff" prompts-those on which every sampled rollout in a group fails-the group-normalized advantage is identically zero, so GRPO produces no gradient on precisely the prompts at the frontier of the model's capability. We introduce LoRA Scaffolded Policy Optimization (LSPO), a sampling-time mechanism that recovers this lost gradient. Each RL step, LSPO detects cliff prompts, fits a small low-rank (LoRA) adapter by a brief supervised step on their ground-truth solutions, re-rolls the cliffs with the base-plus-adapter model, splices the now-successful completions back into the RL batch with an importance-sampling correction, and takes a GRPO step on the base alone; the adapter receives only the supervised gradient and is discarded at checkpoint, yielding a base-only model. On DeepMath-103K with DeepSeek-R1-Distill-Qwen-1.5B, evaluated over n=5 paired seeds per arm at a matched 1000-step reporting horizon, LSPO's 5-seed mean matches or beats a DAPO baseline on all 16 (benchmark, pass@k) cells (15 strict wins and one exact tie), with gains of up to +10.7 points on AIME24/pass@4, +6.7 points on AIME24 and AIME26 at pass@16, and +2.4 points on MATH500/pass@1; averaged over the 16 cells the improvement is +3.8 points.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。