通过推理时搜索优化文本到视频生成的对齐效果,提升画面自然度。
Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search
- 在推理阶段用前瞻估计器搜索更优扩散隐空间,最大化对齐奖励。
- 在不更新模型的前提下,显著提升视觉质量与提示对齐性。
- 适合追求高效高质视频生成的研究者和开发者使用。
文本到视频扩散模型虽能生成逼真视频,但常出现动作不自然、变形或静止等问题。本文提出基于前瞻估计器的扩散隐空间束搜索方法,在推理阶段选择更优隐变量以最大化对齐奖励。我们发现,为提升感知质量,需对现有评估指标加权校准,因传统视频自然度度量与人类及视觉语言模型评价并不总一致。实验表明,该方法在不更新模型参数的情况下,优于贪婪搜索与最佳N采样,计算效率更高,且在校准奖励、视觉语言模型评估和人工打分中均表现最优。结果证明其适用于多种生成模型,并建议将推理资源优先分配给前瞻估计器与扩大搜索预算,而非增加去噪步数。
原文摘要 · Abstract (English)
The remarkable progress in text-to-video diffusion models enables the generation of photorealistic videos, although the content of these generated videos often includes unnatural movement or deformation, reverse playback, and motionless scenes. Recently, an alignment problem has attracted huge attention, where we steer the output of diffusion models based on some measure of the content's goodness. Because there is a large room for improvement of perceptual quality along the frame direction, we should address which metrics we should optimize and how we can optimize them in the video generation. In this paper, we propose diffusion latent beam search with lookahead estimator, which can select a better diffusion latent to maximize a given alignment reward at inference time. We then point out that improving perceptual video quality with respect to alignment to prompts requires reward calibration by weighting existing metrics. This is because when humans or vision language models evaluate outputs, many previous metrics to quantify the naturalness of video do not always correlate with the evaluation. We demonstrate that our method improves the perceptual quality evaluated on the calibrated reward, VLMs, and human assessment, without model parameter update, and outputs the best generation compared to greedy search and best-of-N sampling under much more efficient computational cost. The experiments highlight that our method is beneficial to many capable generative models, and provide a practical guideline: we should prioritize the inference-time compute allocation into enabling the lookahead estimator and increasing the search budget, rather than expanding the denoising steps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。