通过奖励重加权提升数学推理模型训练效率
MMR-GRPO: Accelerating GRPO-Style Training through Diversity-Aware Reward Reweighting
- 用最大边缘相关性动态调整奖励，优先多样解法
- 训练步数减少47.9%，实际耗时降低70.2%
- 适合追求高效训练的AI研究员和工程师
组相对策略优化（GRPO）已成为训练数学推理模型的标准方法，但其依赖每个提示生成多个完成结果，导致计算开销大。尽管近期工作减少了达到峰值性能所需的训练步数，但由于每步成本更高，整体实际训练时间往往未减甚至增加。本文提出MMR-GRPO，引入最大边缘相关性（Maximal Marginal Relevance）根据完成结果的多样性重加权奖励。核心洞察是语义冗余的完成结果带来的边际学习信号有限；优先选择多样解法可获得更丰富的更新信息，加速收敛。在三个模型规模（1.5B、7B、8B）、三种GRPO变体及五个数学推理基准上的广泛评估表明，MMR-GRPO在保持相当峰值性能的同时，平均仅需47.9%的训练步数和70.2%的实际运行时间。该效果在不同模型、方法和数据集上均一致。代码已开源：https://github.com/WeiKangda/MMR-GRPO。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) has become a standard approach for training mathematical reasoning models; however, its reliance on multiple completions per prompt makes training computationally expensive. Although recent work has reduced the number of training steps required to reach peak performance, the overall wall-clock training time often remains unchanged or even increases due to higher per-step cost. We propose MMR-GRPO, which integrates Maximal Marginal Relevance to reweigh rewards based on completion diversity. Our key insight is that semantically redundant completions contribute limited marginal learning signal; prioritizing diverse solutions yields more informative updates and accelerates convergence. Extensive evaluations across three model sizes (1.5B, 7B, 8B), three GRPO variants, and five mathematical reasoning benchmarks show that MMR-GRPO achieves comparable peak performance while requiring on average 47.9% fewer training steps and 70.2% less wall-clock time. These gains are consistent across models, methods, and benchmarks. Our code is released at: https://github.com/WeiKangda/MMR-GRPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。