arXiv:2508.00289cs.CV2025-08ICCV被引 3

无需微调即可精准控制文本生成视频,且内存占用更低。

TITAN-Guide: Taming Inference-Time AligNment for Guided Text-to-Video Diffusion Models

论文配图:TITAN-Guide: Taming Inference-Time AligNment for Guided Text-to-Video Diffusion Models
图 1 · 摘自论文原文
  • 不依赖反向传播,通过前向梯度优化扩散潜空间。
  • 在多个基准上显著提升视频生成质量,同时减少内存消耗。
  • 适合需要高效控制的复杂生成任务,如长视频生成。

当前条件扩散模型在执行特定任务时仍需大量监督微调。利用现成模型进行训练无关的引导是一种避免进一步微调的优选方案。然而,现有训练无关引导框架或内存开销大,或因粗略估计导致控制效果不佳,限制了其在计算密集型任务(如文本到视频扩散模型)中的应用。本文提出TITAN-Guide,一种用于引导文本到视频扩散模型的推理时对齐方法,有效解决内存问题并提升引导精度。我们设计了一种无需反向传播的高效方法,优化扩散潜变量;研究了多种方向性指令下的前向梯度下降策略。实验表明,该方法在潜变量优化过程中显著降低内存使用,优于以往方法。所提方案不仅大幅减少内存需求,还在多个扩散引导基准上显著提升文本到视频生成性能。代码、模型与演示已公开于 https://titanguide.github.io。

原文摘要 · Abstract (English)

In the recent development of conditional diffusion models still require heavy supervised fine-tuning for performing control on a category of tasks. Training-free conditioning via guidance with off-the-shelf models is a favorable alternative to avoid further fine-tuning on the base model. However, the existing training-free guidance frameworks either have heavy memory requirements or offer sub-optimal control due to rough estimation. These shortcomings limit the applicability to control diffusion models that require intense computation, such as Text-to-Video (T2V) diffusion models. In this work, we propose Taming Inference Time Alignment for Guided Text-to-Video Diffusion Model, so-called TITAN-Guide, which overcomes memory space issues, and provides more optimal control in the guidance process compared to the counterparts. In particular, we develop an efficient method for optimizing diffusion latents without backpropagation from a discriminative guiding model. In particular, we study forward gradient descents for guided diffusion tasks with various options on directional directives. In our experiments, we demonstrate the effectiveness of our approach in efficiently managing memory during latent optimization, while previous methods fall short. Our proposed approach not only minimizes memory requirements but also significantly enhances T2V performance across a range of diffusion guidance benchmarks. Code, models, and demo are available at https://titanguide.github.io.

文本生成视频扩散模型推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。