通过消除运动与梯度冗余，实现视频动作迁移的快速生成。
FastVMT: Eliminating Redundancy in Video Motion Transfer
- 用局部注意力掩码减少帧间运动计算冗余
- 重用扩散过程中的梯度，跳过不必要的计算
- 速度提升3.43倍，保持画质与时序一致性
视频动作迁移旨在根据文本提示生成视觉内容，同时转移参考视频中的动作模式。现有方法多采用扩散变换器（DiT）架构，虽尝试加速计算，但未解决结构层面的效率瓶颈。本文识别并消除了两类计算冗余：一是运动冗余——因通用DiT不考虑帧间运动小而平滑；二是梯度冗余——忽略扩散轨迹上梯度变化缓慢的特性。为缓解运动冗余，引入局部注意力掩码，仅在邻近区域计算交互权重；为利用梯度冗余，设计梯度复用优化策略，跳过冗余梯度计算。实验表明，FastVMT平均实现3.43倍加速，且未损害生成视频的视觉保真度与时序一致性。
原文摘要 · Abstract (English)
Video motion transfer aims to synthesize videos by generating visual content according to a text prompt while transferring the motion pattern observed in a reference video. Recent methods predominantly use the Diffusion Transformer (DiT) architecture. To achieve satisfactory runtime, several methods attempt to accelerate the computations in the DiT, but fail to address structural sources of inefficiency. In this work, we identify and remove two types of computational redundancy in earlier work: motion redundancy arises because the generic DiT architecture does not reflect the fact that frame-to-frame motion is small and smooth; gradient redundancy occurs if one ignores that gradients change slowly along the diffusion trajectory. To mitigate motion redundancy, we mask the corresponding attention layers to a local neighborhood such that interaction weights are not computed unnecessarily distant image regions. To exploit gradient redundancy, we design an optimization scheme that reuses gradients from previous diffusion steps and skips unwarranted gradient computations. On average, FastVMT achieves a 3.43x speedup without degrading the visual fidelity or the temporal consistency of the generated videos.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。