用一步优化提升视频生成对齐效率，节省算力且更稳定。
Flash-GRPO: Efficient Alignment for Video Diffusion via One-Step Policy Optimization
- 一步式策略优化，避免分步训练的梯度不稳问题。
- 140亿参数模型训练耗时减少至百GPU天以下，效果更优。
- 适合追求高效训练视频扩散模型的研究者和开发者。
组相对策略优化已成为对齐视频扩散模型与人类偏好的关键方法，但存在严重计算瓶颈：训练一个140亿参数模型通常需要数百个GPU天。现有高效方法通过滑动窗口子采样训练时间步来降低成本，但从根本上损害优化过程，导致严重不稳定且无法达到完整轨迹性能。我们提出Flash-GRPO，一种单步训练框架，在低计算预算下实现了比全轨迹训练更优的对齐质量，同时显著提升训练效率。Flash-GRPO解决两大关键挑战：等时分组通过强制提示级时间一致性，消除时间步混淆方差，使策略性能与时间步难度解耦；时间梯度校正消除了随时间变化的缩放因子，缓解不同时间步间梯度幅值差异过大的问题。在13亿至140亿参数模型上的实验验证了Flash-GRPO的有效性，展现出显著的训练加速、一致的稳定性以及业界领先的对齐质量。
原文摘要 · Abstract (English)
Group Relative Policy Optimization has emerged as essential for aligning video diffusion models with human preferences, but faces a critical computational bottleneck: training a 14B parametered model typically demands hundreds of GPU days per experiment. Existing efficiency methods reduce costs through sliding window subsampling training timesteps, but fundamentally compromise optimization, exhibiting severe instability and failing to reach full trajectory performance. We present Flash-GRPO, a single-step training framework that outperforms full trajectory training in alignment quality under low computational budgets while substantially improving training efficiency. Flash-GRPO addresses two critical challenges: iso-temporal grouping eliminates timestep-confounded variance by enforcing prompt-wise temporal consistency, decoupling policy performance from timestep difficulty; temporal gradient rectification neutralizes the time-dependent scaling factor that causes vastly inconsistent gradient magnitudes across timesteps. Experiments on 1.3B to 14B parameter models validate Flash-GRPO's effectiveness, demonstrating substantial training acceleration with consistent stability and state-of-the-art alignment quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。