让文本生成视频的运动更灵活可控,且无需逐个优化。
EfficientMT: Efficient Temporal Adaptation for Motion Transfer in Text-to-Video Diffusion Models
- 用少量合成数据微调预训练模型,实现端到端运动迁移。
- 在不需测试时优化的前提下,保持高运动控制精度。
- 适合需要快速生成可控动作视频的研究与应用。
生成模型的发展推动了文本到视频(T2V)生成技术的进步,但生成视频的运动可控性仍受限。现有运动迁移方法依赖参考视频的运动表示来引导生成,但通常采用样本特定的优化策略,计算开销大。本文提出 EfficientMT,一种新颖高效的端到端视频运动迁移框架。通过利用少量自动生成的成对运动迁移样本,EfficientMT 能有效将预训练 T2V 模型转化为通用运动迁移框架,准确捕捉并复现多样化的运动模式。具体地,我们复用 T2V 模型主干提取参考视频的时间信息,并提出一个缩放模块以提炼运动相关特征。随后引入时间融合机制,将参考运动特征无缝集成至视频生成过程。在自收集的合成成对样本上训练后,EfficientMT 实现无需测试时优化的通用视频运动迁移。大量实验表明,该方法在效率上优于现有方法,同时保持灵活的运动控制能力。代码将公开于 https://github.com/PrototypeNx/EfficientMT。
原文摘要 · Abstract (English)
The progress on generative models has led to significant advances on text-to-video (T2V) generation, yet the motion controllability of generated videos remains limited. Existing motion transfer methods explored the motion representations of reference videos to guide generation. Nevertheless, these methods typically rely on sample-specific optimization strategy, resulting in high computational burdens. In this paper, we propose EfficientMT, a novel and efficient end-to-end framework for video motion transfer. By leveraging a small set of synthetic paired motion transfer samples, EfficientMT effectively adapts a pretrained T2V model into a general motion transfer framework that can accurately capture and reproduce diverse motion patterns. Specifically, we repurpose the backbone of the T2V model to extract temporal information from reference videos, and further propose a scaler module to distill motion-related information. Subsequently, we introduce a temporal integration mechanism that seamlessly incorporates reference motion features into the video generation process. After training on our self-collected synthetic paired samples, EfficientMT enables general video motion transfer without requiring test-time optimization. Extensive experiments demonstrate that our EfficientMT outperforms existing methods in efficiency while maintaining flexible motion controllability. Our code will be available https://github.com/PrototypeNx/EfficientMT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。