通过难度感知策略和多维度题目重写,提升大模型数学推理能力
Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation
- 设计难度感知的强化学习算法,解决难题更新幅度小的问题
- 通过多角度重写题目,显著提升数据难度且保持答案正确性
- 适合追求高阶数学推理性能的AI研究者与开发者
强化学习结合可验证奖励(RLVR)为提升大模型数学推理能力提供了有效机制。然而,现有方法在算法与数据层面均忽视了更难问题的重要性。算法上,广泛使用的组相对策略优化(GRPO)存在隐性不平衡,难题的策略更新幅度偏低;数据上,增强方法多侧重改写以增加多样性,但未系统提升内在难度。为此,我们提出双重视角的MathForge框架,包含难度感知组策略优化(DGPO)与多维度题目重写(MQR)策略。DGPO通过难度平衡的组优势估计修正GRPO的不平衡,并基于难度加权优先处理难题;MQR则从多个方面重写题目,在保持原始正确答案的前提下提升难度。整体形成协同闭环:MQR拓展数据边界,DGPO高效学习增强数据。大量实验表明,MathForge在多种数学推理任务中显著优于现有方法。代码与增强数据已公开于https://github.com/AMAP-ML/MathForge。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) offers a robust mechanism for enhancing mathematical reasoning in large models. However, we identify a systematic lack of emphasis on more challenging questions in existing methods from both algorithmic and data perspectives, despite their importance for refining underdeveloped capabilities. Algorithmically, widely used Group Relative Policy Optimization (GRPO) suffers from an implicit imbalance where the magnitude of policy updates is lower for harder questions. Data-wise, augmentation approaches primarily rephrase questions to enhance diversity without systematically increasing intrinsic difficulty. To address these issues, we propose a two-dual MathForge framework to improve mathematical reasoning by targeting harder questions from both perspectives, which comprises a Difficulty-Aware Group Policy Optimization (DGPO) algorithm and a Multi-Aspect Question Reformulation (MQR) strategy. Specifically, DGPO first rectifies the implicit imbalance in GRPO via difficulty-balanced group advantage estimation, and further prioritizes harder questions by difficulty-aware question-level weighting. Meanwhile, MQR reformulates questions across multiple aspects to increase difficulty while maintaining the original gold answer. Overall, MathForge forms a synergistic loop: MQR expands the data frontier, and DGPO effectively learns from the augmented data. Extensive experiments show that MathForge significantly outperforms existing methods on various mathematical reasoning tasks. The code and augmented data are all available at https://github.com/AMAP-ML/MathForge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。