解决大模型推理中的思考-答案错位问题，提升训练稳定性。
Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweighting
- 基于噪声感知的优化策略，动态调整奖励权重以减少干扰信号影响。
- 在数学推理任务上，性能较基线提升2.2%至2.5%，且在20%噪声下仍能稳定训练。
- 适合追求高鲁棒性推理模型训练的研究者与工程团队。
组相对策略优化（GRPO）是训练大型推理模型的关键技术，但存在严重缺陷：即‘思考-答案错位’问题，其中噪声奖励信号会污染学习过程。该问题在响应组不平衡时尤为严重，反而在最需要可靠信号时使信号质量下降。为此，我们提出稳定的组相对策略优化（S-GRPO），一种原理性改进方法，通过推导出最优、噪声感知的优势权重来稳定训练。我们在数学推理基准上的全面实验表明，S-GRPO具有显著有效性与鲁棒性。在多种模型上，其表现优于标准GRPO：Qwen-Math-7B-Base提升2.5%，Llama-3.2-3B-Base提升2.2%，Qwen-Math-1.5B-Instruct提升2.4%。最关键的是，当标准GRPO在20%合成奖励噪声下无法学习时，S-GRPO仍保持稳定的学习进度。这些结果凸显了S-GRPO在更鲁棒、高效训练大规模推理模型方面的潜力。
原文摘要 · Abstract (English)
Group-Relative Policy Optimization (GRPO) is a key technique for training large reasoning models, yet it suffers from a critical vulnerability: the \emph{Think-Answer Mismatch}, where noisy reward signals corrupt the learning process. This problem is most severe in unbalanced response groups, paradoxically degrading the signal precisely when it should be most informative. To address this challenge, we propose Stable Group-Relative Policy Optimization (S-GRPO), a principled enhancement that derives optimal, noise-aware advantage weights to stabilize training. Our comprehensive experiments on mathematical reasoning benchmarks demonstrate S-GRPO's effectiveness and robustness. On various models, S-GRPO significantly outperforms DR. GRPO, achieving performance gains of +2.5% on Qwen-Math-7B-Base, +2.2% on Llama-3.2-3B-Base, and +2.4% on Qwen-Math-1.5B-Instruct. Most critically, while standard GRPO fails to learn under 20% synthetic reward noise, S-GRPO maintains stable learning progress. These results highlight S-GRPO's potential for more robust and effective training of large-scale reasoning models. \footnote{Code and data are available at: https://github.com/shenpeijun0212/S-GRPO
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。