arXiv:2603.17398cs.CV2026-03

让视频生成更轻量,根据运动动态调整注意力。

Motion-Adaptive Temporal Attention for Lightweight Video Generation with Stable Diffusion

  • 按运动强度动态调节帧间注意力范围,高动作用局部关注,低动作用全局关注。
  • 仅增加2580万参数(占原模型2.9%),在10万视频上训练后效果媲美主流方法。
  • 无需额外一致性损失,靠去噪目标隐式实现时序稳定,适合高效视频生成场景。

我们提出一种面向冻结Stable Diffusion模型的轻量化视频生成运动自适应时序注意力机制。不同于统一处理所有视频内容,该方法根据估计的运动信息动态调整时序注意力感受野:高运动序列采用局部跨帧关注以保留快速变化细节,低运动序列则采用全局关注以保证场景一致性。通过级联策略将轻量级时序注意力模块注入UNet Transformer的所有块中——下采样与中间块使用全局注意力实现语义稳定,上采样块使用运动自适应注意力进行精细优化。结合时序相关噪声初始化与运动感知门控,系统仅引入2580万可训练参数(占基础UNet的2.9%),在10万视频上训练后,在WebVid验证集上达到具有竞争力的结果。实验表明,标准去噪目标本身已提供充分的隐式时序正则化,优于添加显式时序一致性损失的方法。消融研究揭示了噪声相关性与运动幅度之间的明确权衡,为推理阶段提供了多样生成行为的实用控制手段。

原文摘要 · Abstract (English)

We present a motion-adaptive temporal attention mechanism for parameter-efficient video generation built upon frozen Stable Diffusion models. Rather than treating all video content uniformly, our method dynamically adjusts temporal attention receptive fields based on estimated motion content: high-motion sequences attend locally across frames to preserve rapidly changing details, while low-motion sequences attend globally to enforce scene consistency. We inject lightweight temporal attention modules into all UNet transformer blocks via a cascaded strategy -- global attention in down-sampling and middle blocks for semantic stabilization, motion-adaptive attention in up-sampling blocks for fine-grained refinement. Combined with temporally correlated noise initialization and motion-aware gating, the system adds only 25.8M trainable parameters (2.9\% of the base UNet) while achieving competitive results on WebVid validation when trained on 100K videos. We demonstrate that the standard denoising objective alone provides sufficient implicit temporal regularization, outperforming approaches that add explicit temporal consistency losses. Our ablation studies reveal a clear trade-off between noise correlation and motion amplitude, providing a practical inference-time control for diverse generation behaviors.

视频生成扩散模型轻量化注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。