arXiv:2605.14988cs.CV2026-05

用推理阶段引导提升文本生成视频的组合理解能力。

Compositional Video Generation via Inference-Time Guidance

论文配图:Compositional Video Generation via Inference-Time Guidance
图 1 · 摘自论文原文
  • 利用模型内部注意力图捕捉概念时空关联,实现无重训练引导。
  • 在三个基准上显著提升组合忠实度,视觉质量保持不变。
  • 适合需要精准语义控制的视频生成应用,无需额外标注。

文本到视频的扩散模型能生成逼真视频,但在需要细粒度组合理解的任务上表现不佳,如实体间关系、属性、动作及运动方向等。我们提出在推理阶段通过模型自身内部的定位信号来引导去噪过程,而非重新训练生成器。关键观察是:跨注意力图已编码提示概念在时空上的定位信息。我们在这些注意力特征上训练一个轻量级组合分类器,并在早期去噪步骤中使用其梯度来引导潜在轨迹向期望的组合方向发展。该方法基于冻结的视觉语言模型骨干网络,分类器可跨语义相关组合标签迁移,不依赖特定类别特征。CVG在不修改模型架构、不微调生成器、无需布局或边界框等用户输入的前提下,提升了组合生成的忠实度。在多个组合文本到视频基准测试中,显著提升了提示忠实度,同时保持了底层生成器的视觉质量。

原文摘要 · Abstract (English)

Text-to-video diffusion models generate realistic videos, but often fail on prompts requiring fine-grained compositional understanding, such as relations between entities, attributes, actions, and motion directions. We hypothesize that these failures need not be addressed by retraining the generator, but can instead be mitigated by steering the denoising process using the model's own internal grounding signals. We propose \textbf{CVG}, an inference-time guidance method for improving compositional faithfulness in frozen text-to-video models. Our key observation is that cross-attention maps already encode how prompt concepts are grounded across space and time. We train a lightweight compositional classifier on these attention features and use its gradients during early denoising steps to steer the latent trajectory toward the desired composition. Built on a frozen VLM backbone, the classifier transfers across semantically related composition labels rather than relying only on narrow category-specific features. CVG improves compositional generation without modifying the model architecture, fine-tuning the generator, or requiring layouts, boxes, or other user-supplied controls. Experiments on compositional text-to-video benchmarks show improved prompt faithfulness while preserving the visual quality of the underlying generator.

视频生成扩散模型推理引导组合理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。