让视频生成强化学习更稳定,通过约束探索空间提升质量与可靠性。
Manifold-Aware Exploration for Reinforcement Learning in Video Generation
- 在微观和宏观层面约束探索,保持生成轨迹贴近数据流形。
- 在HunyuanVideo1.5上,各项指标均优于现有方法,包括CLIPScore和PickScore。
- 适合关注视频生成质量与训练稳定性的研究者与开发者。
视频生成中的群组相对策略优化(GRPO)方法如FlowGRPO仍远不如语言和图像模型可靠,原因在于视频解空间复杂,且从常微分方程(ODE)到随机微分方程(SDE)的转换会引入过多噪声,降低轨迹质量并使奖励估计不可靠,进而破坏对齐训练的稳定性。为此,本文将预训练模型视为定义了有效的视频数据流形,将核心问题建模为在该流形附近进行探索,以保障轨迹质量并维持可靠的奖励估计。提出SAGE-GRPO(通过探索实现稳定对齐),在微观层面设计带对数曲率修正的流形感知SDE,并引入梯度范数均衡器,稳定跨时间步采样与更新;在宏观层面采用双信任域机制,结合周期性移动锚点与分步约束,使信任域追踪更接近流形的检查点,抑制长时程漂移。在HunyuanVideo1.5上使用原始VideoAlign作为奖励模型评估,结果显示在视频质量(VQ)、多尺度质量(MQ)、时间一致性(TA)及视觉指标(CLIPScore、PickScore)上持续领先于先前方法,验证了其在奖励最大化与整体视频质量上的优越性。代码与可视化图集见https://dungeonmassster.github.io/SAGE-GRPO-Page/。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) methods for video generation like FlowGRPO remain far less reliable than their counterparts for language models and images. This gap arises because video generation has a complex solution space, and the ODE-to-SDE conversion used for exploration can inject excess noise, lowering rollout quality and making reward estimates less reliable, which destabilizes post-training alignment. To address this problem, we view the pre-trained model as defining a valid video data manifold and formulate the core problem as constraining exploration within the vicinity of this manifold, ensuring that rollout quality is preserved and reward estimates remain reliable. We propose SAGE-GRPO (Stable Alignment via Exploration), which applies constraints at both micro and macro levels. At the micro level, we derive a precise manifold-aware SDE with a logarithmic curvature correction and introduce a gradient norm equalizer to stabilize sampling and updates across timesteps. At the macro level, we use a dual trust region with a periodic moving anchor and stepwise constraints so that the trust region tracks checkpoints that are closer to the manifold and limits long-horizon drift. We evaluate SAGE-GRPO on HunyuanVideo1.5 using the original VideoAlign as the reward model and observe consistent gains over previous methods in VQ, MQ, TA, and visual metrics (CLIPScore, PickScore), demonstrating superior performance in both reward maximization and overall video quality. The code and visual gallery are available at https://dungeonmassster.github.io/SAGE-GRPO-Page/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。