动态剪枝加速大模型推理训练，不偏移优化目标。
Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization
- 用重要性采样修正剪枝偏差，保持梯度无偏。
- 训练速度提升2.37倍，数学推理准确率高3.36%。
- 适合大规模语言模型高效训练场景。
组相对策略优化（GRPO）能有效扩展大模型推理能力，但因大量分组采样导致计算成本过高。现有选择性数据使用方法虽可缓解开销，却会改变采样分布，引发估计偏差，影响理论严谨性和收敛性。为此，我们提出动态剪枝策略优化（DPPO），通过基于重要性采样的校正机制，在实现动态剪枝的同时保持无偏梯度估计。结合数学推导的重缩放因子，DPPO显著加速GRPO训练，且不改变全批量基线的优化目标。此外，为缓解剪枝带来的数据稀疏问题，我们引入密集提示打包（Dense Prompt Packing），一种基于窗口的贪心策略，最大化有效标记密度与硬件利用率。大量实验表明，DPPO在多种模型与基准上持续加速训练。例如，在Qwen3-4B模型上训练MATH数据集时，实现2.37倍训练速度提升，并在六个数学推理基准上平均准确率超出GRPO 3.36%。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) effectively scales LLM reasoning but incurs prohibitive computational costs due to its extensive group-based sampling requirement. While recent selective data utilization methods can mitigate this overhead, they could induce estimation bias by altering the underlying sampling distribution, compromising theoretical rigor and convergence behavior. To address this limitation, we propose Dynamic Pruning Policy Optimization (DPPO), a framework that enables dynamic pruning while preserving unbiased gradient estimation through importance sampling-based correction. By incorporating mathematically derived rescaling factors, DPPO significantly accelerates GRPO training without altering the optimization objective of the full-batch baseline. Furthermore, to mitigate the data sparsity induced by pruning, we introduce Dense Prompt Packing, a window-based greedy strategy that maximizes valid token density and hardware utilization. Extensive experiments demonstrate that DPPO consistently accelerates training across diverse models and benchmarks. For instance, on Qwen3-4B trained on MATH, DPPO achieves 2.37$\times$ training speedup and outperforms GRPO by 3.36% in average accuracy across six mathematical reasoning benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。