让难问题持续获得梯度信号，提升模型推理能力。
Max Out GRPO Signal: Adaptive Trace Prefix Control for Hard Reasoning Problems
- 动态调节解题前缀长度，维持成功率为50%以最大化梯度信号
- 0.6B模型在数学题上准确率提升2.1倍，其他模型增益1.6~1.7倍
- 无需修改训练器，仅通过数据预处理和损失掩码即可实现
组相对策略优化（GRPO）在模型最难的问题上失效：当一组采样中无一成功时，组相对优势消失，导致无法提供梯度，浪费了最值得学习的难题。通过在问题前添加参考解的正确前缀，可提升成功率，使前缀长度成为连续控制难度的旋钮。现有方法仅固定设置一次；而AdaPrefix-GRPO将其变为反馈控制器，在训练全程动态调整每道题获得的前缀长度，将成功率稳定在50%附近（此时GRPO梯度最大），随后完全撤除辅助，使部署模型独立解决。在硬核数学任务上，相同训练算力下，0.6B模型在分布外问题上的准确率较GRPO提升2.1倍，Qwen3-1.7B模型提升1.6倍，AIME任务提升1.7倍，同时将轨迹长度大致减半。该方法仅需数据准备阶段添加前缀，并对前缀令牌施加损失掩码，训练器本身无需改动。模型越小，收益越大。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) stalls on a model's hardest problems: when no rollout in a group succeeds, the group-relative advantages vanish and the problem contributes no gradient, wasting the frontier examples we most want to learn from. Prepending a correct prefix of a reference solution raises the success rate, making prefix length a continuous knob on difficulty. Concurrent methods set the knob once; AdaPrefix-GRPO turns it into a feedback controller: throughout training it adjusts how much of the solution each problem gets, holding its success rate near 50%, where GRPO's gradient signal is largest, then withdraws the assistance entirely, so the deployed model solves problems unaided. On hard math, at matched training FLOPs, it more than doubles GRPO's accuracy on held-out problems from the training distribution for a 0.6B model (2.1x), with 1.6x on Qwen3-1.7B and 1.7x on AIME, while roughly halving trace length. The method is implemented in data preparation plus a loss mask on prefix tokens; the trainer is otherwise stock. The smaller the model, the larger the gain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。