提出动态剪枝方法,提升生成模型对齐效率与多样性。
Expand and Prune: Maximizing Trajectory Diversity for Effective GRPO in Generative Models
- 采样时基于潜在特征提前剔除冗余轨迹,降低计算开销。
- 通过扩增再剪枝策略,使轨迹多样性提升37%以上。
- 适用于扩散与流模型,尤其适合资源受限场景。
群体相对策略优化(GRPO)虽能有效对齐生成模型,但受限于大组规模带来的高昂计算成本。本文通过实证研究发现:大量轨迹会聚集到组均奖励附近,导致优化价值有限。为此,我们提出最优方差过滤(OVF)策略,证明高方差子集优于全量组。然而静态的后采样过滤仍需冗余计算。为此,我们设计主动式GRPO(Pro-GRPO),将潜在空间轨迹剪枝嵌入采样过程,实现早期终止冗余轨迹。利用其高效性,采用“扩增-剪枝”策略:先扩大初始采样组以增强多样性,再通过多步OVF处理潜在表示,避免高成本。在扩散与流模型上的广泛实验验证了该框架的通用性与有效性。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) is a powerful technique for aligning generative models, but its effectiveness is bottlenecked by the conflict between large group sizes and prohibitive computational costs. In this work, we investigate the trade-off through empirical studies, yielding two key observations. First, we discover the reward clustering phenomenon in which many trajectories collapse toward the group-mean reward, offering limited optimization value. Second, we design a heuristic strategy named Optimal Variance Filtering (OVF), and verify that a high-variance subset of trajectories, selected by OVF can outperform the larger, unfiltered group. However, this static, post-sampling OVF approach still necessitates critical computational overhead, as it performs unnecessary sampling for trajectories that are ultimately discarded. To resolve this, we propose Pro-GRPO (Proactive GRPO), a novel dynamic framework that integrates latent feature-based trajectory pruning into the sampling process. Through the early termination of reward-clustered trajectories, Pro-GRPO reduces computational overhead. Leveraging its efficiency, Pro-GRPO employs an "Expand-and-Prune" strategy. This strategy first expands the size of initial sampling group to maximize trajectory diversity, then it applies multi-step OVF to the latents, avoiding prohibitive computational costs. Extensive experiments on both diffusion-based and flow-based models demonstrate the generality and effectiveness of our Pro-GRPO framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。