让大模型解数学题更简洁准确，自动识别题目难易并优化推理过程。
GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models
- 引入长度正则化奖励，强制输出简洁答案
- 对错误解答施加显式惩罚，提升解题精度
- 根据题目难度重加权优势值，增强复杂问题泛化能力
Group Relative Policy Optimization（GRPO）被广泛用于类似R1的推理模型，显著提升了数学推理能力。然而，GRPO仍面临奖励稀疏、输出冗长和对问题难度关注不足的问题。本文提出GRPO-LEAD，通过三项改进：(1) 长度正则化奖励，鼓励简洁表达同时保持准确性；(2) 对错误解法施加显式惩罚，提高模型精确性；(3) 基于难度感知的优势重加权，增强在难题上的泛化能力。全面评估表明，GRPO-LEAD显著提升推理准确率、简洁性和效率。该方法在14B规模模型上达到当前最优性能，凸显其与合适模型规模及高质量数据的协同效应。代码、生成数据集及模型已开源。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO), which is widely adopted by R1-like reasoning models, has advanced mathematical reasoning. Nevertheless, GRPO faces challenges in reward sparsity, verbosity, and inadequate focus on problem difficulty. We propose GRPO-LEAD, enhancing GRPO with: (1) length-regularized rewards to encourage conciseness while maintaining accuracy; (2) explicit penalties for incorrect solutions to improve model precision; and (3) difficulty-aware advantage reweighting for robust generalization on challenging problems. Comprehensive evaluations demonstrate that GRPO-LEAD significantly improves reasoning accuracy, conciseness, and efficiency. Our approach achieves state-of-the-art performance for 14B-scale models, underscoring the synergy of our methods with appropriate model scale and high-quality data. Our source code, generated dataset, and models are available at https://github.com/aeroplanepaper/GRPO-LEAD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。