通过剪枝低效完成项，加速推理模型训练。
CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models
- 只保留相对优势高的完成项进行梯度更新，减少计算量。
- 在GSM8K上提速7.98倍，Math上提速3.48倍，准确率不降反升。
- 适合需要高效训练推理模型的研究者和开发者。
本文提出完成项剪枝策略（CPPO），用于加速基于组相对策略优化（GRPO）的推理模型训练。尽管GRPO效果显著，但需对每道题采样多个完成项，导致训练成本高昂。实验与理论分析表明，完成项数量影响模型准确率，且训练时间随其数量呈倍数增长，而并非所有完成项对策略训练的贡献均等，其贡献取决于相对优势。为此，我们提出CPPO，通过剪枝绝对优势较低的完成项，显著减少梯度计算所需样本数量。此外，引入动态完成项分配策略，通过增加问题数量提升GPU利用率，进一步提高训练效率。实验结果显示，相较于原始GRPO，CPPO在GSM8K上实现最高7.98倍加速，在Math数据集上实现3.48倍加速，同时保持甚至提升模型准确率。代码已开源：https://github.com/lzhxmu/CPPO。
原文摘要 · Abstract (English)
This paper introduces Completion Pruning Policy Optimization (CPPO) to accelerate the training of reasoning models based on Group Relative Policy Optimization (GRPO). GRPO, while effective, incurs high training costs due to the need to sample multiple completions for each question. Our experiment and theoretical analysis reveal that the number of completions impacts model accuracy yet increases training time multiplicatively, and not all completions contribute equally to policy training -- their contribution depends on their relative advantage. To address these issues, we propose CPPO, which prunes completions with low absolute advantages, significantly reducing the number needed for gradient calculation and updates. Additionally, we introduce a dynamic completion allocation strategy to maximize GPU utilization by incorporating additional questions, further enhancing training efficiency. Experiments show that CPPO achieves up to $7.98\times$ speedup on GSM8K and $3.48\times$ on Math while preserving or even enhancing the accuracy compared to the original GRPO. We release our code at \href{https://github.com/lzhxmu/CPPO}{https://github.com/lzhxmu/CPPO}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。