arXiv:2602.04928cs.LG2026-02被引 2

通过过程奖励梯度引导生成过程,实现视频生成的高效对齐与加速。

Euphonium: Steering Video Flow Matching via Process Reward Gradient Guided Stochastic Dynamics

  • 将采样过程建模为带奖励梯度驱动的随机微分方程,实现每一步的精准引导。
  • 在文本到视频生成中训练速度提升1.66倍,且生成结果更符合人类偏好。
  • 适用于需要快速高质量视频生成的场景,尤其适合偏好对齐任务。

尽管在线强化学习已成为对齐流匹配模型与人类偏好的关键技术,但现有方法在训练回放过程中探索效率低下。其依赖无方向的随机性和稀疏的最终奖励,难以发现高奖励样本,导致数据利用效率低且优化缓慢。为此,我们提出Euphonium,一种通过过程奖励梯度引导动态的新型框架。核心思想是将采样过程建模为理论完备的随机微分方程,显式引入过程奖励模型的梯度作为流动漂移项。该设计实现密集、逐步的高奖励区域引导,超越了以往方法的无向探索,并理论上涵盖现有方法(如Flow-GRPO、DanceGRPO)作为特例。我们进一步推导出一种蒸馏目标,将引导信号内化至流网络,消除推理时对奖励模型的依赖。我们以双奖励组相对策略优化算法实例化该框架,结合潜在空间过程奖励实现高效信用分配,以及像素级结果奖励保证最终视觉保真度。在文本到视频生成任务上的实验表明,Euphonium相比现有方法实现了更优的对齐效果,同时训练收敛速度加快1.66倍。代码已开源:https://github.com/zerzerzerz/Euphonium。

原文摘要 · Abstract (English)

While online Reinforcement Learning has emerged as a crucial technique for aligning flow matching models with human preferences, current approaches are hindered by inefficient exploration during training rollouts. Relying on undirected stochasticity and sparse outcome rewards, these methods struggle to discover high-reward samples, resulting in data-inefficient and slow optimization. To address these limitations, we propose Euphonium, a novel framework that steers generation via process reward gradient guided dynamics. Our key insight is to formulate the sampling process as a theoretically principled Stochastic Differential Equation that explicitly incorporates the gradient of a Process Reward Model into the flow drift. This design enables dense, step-by-step steering toward high-reward regions, advancing beyond the unguided exploration in prior works, and theoretically encompasses existing sampling methods (e.g., Flow-GRPO, DanceGRPO) as special cases. We further derive a distillation objective that internalizes the guidance signal into the flow network, eliminating inference-time dependency on the reward model. We instantiate this framework with a Dual-Reward Group Relative Policy Optimization algorithm, combining latent process rewards for efficient credit assignment with pixel-level outcome rewards for final visual fidelity. Experiments on text-to-video generation show that Euphonium achieves better alignment compared to existing methods while accelerating training convergence by 1.66x. Our code is available at https://github.com/zerzerzerz/Euphonium

视频生成强化学习流匹配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。