改进视频扩散模型的运动迁移能力,实现动作与外观解耦。
Decouple and Track: Benchmarking and Improving Video Diffusion Transformers for Motion Transfer
- 引入时序核平滑特征,分离运动与外观信息。
- 在潜空间对密集轨迹显式监督,提升运动一致性。
- 提出新基准与混合指标,更全面评估迁移效果。
运动迁移任务旨在将源视频中的动作迁移到新生成视频中,要求模型将运动与外观解耦。以往基于扩散的方法主要依赖3D U-Net中的独立时空注意力机制,而当前最先进的视频扩散变压器(DiT)模型采用3D全注意力,未显式分离时空信息,导致运动与外观耦合更难。本文提出DeT方法,通过引入简单有效的时序核,在时间维度上平滑DiT特征,促进前景运动与背景外观的解耦;同时,该时序核有效捕捉特征中的时间变化,与运动密切相关。此外,我们在潜空间沿密集轨迹引入显式监督,进一步增强运动一致性。我们还提出了MTBench,一个通用且具有挑战性的运动迁移基准,并设计了结合全局与局部运动相似性的混合运动保真度度量。大量实验表明,DeT在运动保真度与编辑保真度之间取得了最佳平衡。
原文摘要 · Abstract (English)
The motion transfer task aims to transfer motion from a source video to newly generated videos, requiring the model to decouple motion from appearance. Previous diffusion-based methods primarily rely on separate spatial and temporal attention mechanisms within the 3D U-Net. In contrast, state-of-the-art video Diffusion Transformers (DiT) models use 3D full attention, which does not explicitly separate temporal and spatial information. Thus, the interaction between spatial and temporal dimensions makes decoupling motion and appearance more challenging for DiT models. In this paper, we propose DeT, a method that adapts DiT models to improve motion transfer ability. Our approach introduces a simple yet effective temporal kernel to smooth DiT features along the temporal dimension, facilitating the decoupling of foreground motion from background appearance. Meanwhile, the temporal kernel effectively captures temporal variations in DiT features, which are closely related to motion. Moreover, we introduce explicit supervision along dense trajectories in the latent feature space to further enhance motion consistency. Additionally, we present MTBench, a general and challenging benchmark for motion transfer. We also introduce a hybrid motion fidelity metric that considers both the global and local motion similarity. Therefore, our work provides a more comprehensive evaluation than previous works. Extensive experiments on MTBench demonstrate that DeT achieves the best trade-off between motion fidelity and edit fidelity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。