arXiv:2504.15032cs.CV2025-04被引 5

无需训练即可精准控制视频生成中的物体布局与动态交互。

DyST-XL: Dynamic Layout Planning and Content Control for Compositional Text-to-Video Generation

  • 用大模型解析提示词生成物理合理的关键帧布局,中间帧通过轨迹优化填补。
  • 通过帧级注意力掩码实现每个物体的精确控制,避免位置错乱。
  • 通过首帧特征传播保持物体身份一致,适合复杂场景生成任务。

组合式文本到视频生成要求合成包含多个互动实体及精确时空关系的动态场景,这对基于扩散模型的方法仍是一大挑战。现有方法因缺乏约束的交叉注意力机制和不足的物理感知推理,常出现布局不连续、实体身份漂移和不合理交互动态。为此,我们提出 DyST-XL,一个无需训练的框架,通过帧级控制增强现成文本到视频模型(如 CogVideoX-5B)。该框架集成三项核心创新:(1) 动态布局规划器利用大语言模型(LLMs)将输入提示解析为实体属性图,并生成物理感知的关键帧布局,中间帧通过轨迹优化插值;(2) 双提示控制注意力机制通过帧级注意力掩码实现局部文本-视频对齐,精确控制单个实体;(3) 实体一致性约束策略在去噪过程中将首帧特征嵌入传播至后续帧,无需人工标注即可保持物体身份。实验表明,DyST-XL 在复杂提示下表现显著提升,有效填补了无训练视频合成的关键空白。代码已开源:https://github.com/XiaoBuL/DyST-XL。

原文摘要 · Abstract (English)

Compositional text-to-video generation, which requires synthesizing dynamic scenes with multiple interacting entities and precise spatial-temporal relationships, remains a critical challenge for diffusion-based models. Existing methods struggle with layout discontinuity, entity identity drift, and implausible interaction dynamics due to unconstrained cross-attention mechanisms and inadequate physics-aware reasoning. To address these limitations, we propose DyST-XL, a \textbf{training-free} framework that enhances off-the-shelf text-to-video models (e.g., CogVideoX-5B) through frame-aware control. DyST-XL integrates three key innovations: (1) A Dynamic Layout Planner that leverages large language models (LLMs) to parse input prompts into entity-attribute graphs and generates physics-aware keyframe layouts, with intermediate frames interpolated via trajectory optimization; (2) A Dual-Prompt Controlled Attention Mechanism that enforces localized text-video alignment through frame-aware attention masking, achieving precise control over individual entities; and (3) An Entity-Consistency Constraint strategy that propagates first-frame feature embeddings to subsequent frames during denoising, preserving object identity without manual annotation. Experiments demonstrate that DyST-XL excels in compositional text-to-video generation, significantly improving performance on complex prompts and bridging a crucial gap in training-free video synthesis. The code is released in https://github.com/XiaoBuL/DyST-XL.

视频生成文本生成视频动态布局无训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。