用视频生成模型做潜在空间奖励模型,提升对齐效果并大幅降耗。
Video Generation Models Are Good Latent Reward Models
- 在潜在空间直接优化,避免图像解码的高开销。
- 相比像素空间方法,训练时间减少60%,显存降低75%以上。
- 适合追求高效视频生成对齐的研究者和工业应用
奖励反馈学习(ReFL)在图像生成对齐人类偏好方面表现优异,但其扩展至视频生成面临挑战。现有视频奖励模型依赖为像素空间输入设计的视觉-语言模型,导致ReFL优化仅能在近完全去噪阶段进行,且需昂贵的VAE解码,带来显著内存开销和训练时间增加,同时缺乏早期阶段监督,仅优化视觉质量而非运动动态与结构连贯性。本文表明,预训练视频生成模型天然适合作为噪声潜在空间中的奖励模型,因其明确设计用于处理任意时刻的噪声潜在表示,并通过序列建模能力保持时序信息。为此,我们提出过程奖励反馈学习(PRFL),实现完全在潜在空间内的偏好优化,支持在整个去噪链中高效梯度反向传播,无需VAE解码。大量实验表明,PRFL显著提升与人类偏好的对齐程度,同时相较RGB ReFL大幅降低内存消耗和训练时间。
原文摘要 · Abstract (English)
Reward feedback learning (ReFL) has proven effective for aligning image generation with human preferences. However, its extension to video generation faces significant challenges. Existing video reward models rely on vision-language models designed for pixel-space inputs, confining ReFL optimization to near-complete denoising steps after computationally expensive VAE decoding. This pixel-space approach incurs substantial memory overhead and increased training time, and its late-stage optimization lacks early-stage supervision, refining only visual quality rather than fundamental motion dynamics and structural coherence. In this work, we show that pre-trained video generation models are naturally suited for reward modeling in the noisy latent space, as they are explicitly designed to process noisy latent representations at arbitrary timesteps and inherently preserve temporal information through their sequential modeling capabilities. Accordingly, we propose Process Reward Feedback Learning~(PRFL), a framework that conducts preference optimization entirely in latent space, enabling efficient gradient backpropagation throughout the full denoising chain without VAE decoding. Extensive experiments demonstrate that PRFL significantly improves alignment with human preferences, while achieving substantial reductions in memory consumption and training time compared to RGB ReFL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。