arXiv:2509.21839cs.CVcs.AI2025-09被引 5

无需训练即可精准控制视频中物体运动轨迹,适配扩散变换器模型。

DiTraj: training-free trajectory control for video diffusion transformer

  • 用大模型分离前景背景提示,引导视频区域生成
  • 修改前景帧位置编码,增强跨帧注意力与轨迹连贯性
  • 无需训练,直接适配DiT模型,提升生成质量与可控性

基于3D全注意力机制的扩散变换器(DiT)在视频生成任务中表现优异。轨迹控制是可控视频生成中的用户友好型任务,但现有方法要么需要大量训练资源,要么仅针对U-Net设计,未能发挥DiT的性能优势。为此,我们提出DiTraj——一种专为DiT设计的简单高效、无需训练的轨迹控制框架。首先,通过大语言模型(LLM)将用户提示拆分为前景与背景提示,分别指导视频中对应区域的生成;其次,分析3D全注意力机制,发现跨标记注意力分数与位置嵌入存在强关联,据此提出时空解耦的3D-RoPE(STD-RoPE),仅修改前景标记的位置嵌入,消除其跨帧空间偏差,增强跨帧注意力,从而强化轨迹控制;此外,通过调节位置嵌入密度实现3D感知的轨迹控制。大量实验表明,该方法在视频质量和轨迹可控性上均优于已有方法。

原文摘要 · Abstract (English)

Diffusion Transformers (DiT)-based video generation models with 3D full attention exhibit strong generative capabilities. Trajectory control represents a user-friendly task in the field of controllable video generation. However, existing methods either require substantial training resources or are specifically designed for U-Net, do not take advantage of the superior performance of DiT. To address these issues, we propose DiTraj, a simple but effective training-free framework for trajectory control in text-to-video generation, tailored for DiT. Specifically, first, to inject the object's trajectory, we propose foreground-background separation guidance: we use the Large Language Model (LLM) to convert user-provided prompts into foreground and background prompts, which respectively guide the generation of foreground and background regions in the video. Then, we analyze 3D full attention and explore the tight correlation between inter-token attention scores and position embedding. Based on this, we propose inter-frame Spatial-Temporal Decoupled 3D-RoPE (STD-RoPE). By modifying only foreground tokens' position embedding, STD-RoPE eliminates their cross-frame spatial discrepancies, strengthening cross-frame attention among them and thus enhancing trajectory control. Additionally, we achieve 3D-aware trajectory control by regulating the density of position embedding. Extensive experiments demonstrate that our method outperforms previous methods in both video quality and trajectory controllability.

视频生成扩散模型轨迹控制DiT

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。