提出高效视频动作迁移框架,解决大模型动作不连贯与训练慢问题。
Follow-Your-Motion: Video Motion Transfer via Efficient Spatial-Temporal Decoupled Finetuning
- 分离时空注意力结构,用解耦LoRA提升动作一致性
- 稀疏采样+自适应RoPE,使微调速度显著提升
- 构建MotionBench基准,覆盖复杂动作类型
近期,视频扩散变换器在多样动作生成方面取得显著进展。针对动作迁移任务,现有方法多采用两阶段低秩适应(LoRA)微调以获得更好性能。然而,现有基于适配的方法在大型视频扩散变换器上仍面临动作不一致和调优效率低的问题。由于3D注意力算子中固有的时空耦合性,简单两阶段LoRA微调难以保持生成视频与输入视频之间的动作一致性,且两阶段均需耗时的微调过程。为此,我们提出Follow-Your-Motion,一种高效的两阶段视频动作迁移框架,可微调强大的视频扩散变换器以合成复杂动作。具体而言,我们提出时空解耦的LoRA,解耦注意力架构中的空间外观与时间动作处理。在第二阶段训练中,设计稀疏动作采样与自适应旋转位置编码(adaptive RoPE),加速微调速度。为弥补该领域缺乏基准的不足,我们引入MotionBench,一个涵盖多种动作类型的综合性基准,包括创意镜头运动、单物体运动、多物体运动及复杂人体动作。我们在MotionBench上进行了广泛评估,验证了Follow-Your-Motion的优越性。
原文摘要 · Abstract (English)
Recently, breakthroughs in the video diffusion transformer have shown remarkable capabilities in diverse motion generations. As for the motion-transfer task, current methods mainly use two-stage Low-Rank Adaptations (LoRAs) finetuning to obtain better performance. However, existing adaptation-based motion transfer still suffers from motion inconsistency and tuning inefficiency when applied to large video diffusion transformers. Naive two-stage LoRA tuning struggles to maintain motion consistency between generated and input videos due to the inherent spatial-temporal coupling in the 3D attention operator. Additionally, they require time-consuming fine-tuning processes in both stages. To tackle these issues, we propose Follow-Your-Motion, an efficient two-stage video motion transfer framework that finetunes a powerful video diffusion transformer to synthesize complex motion. Specifically, we propose a spatial-temporal decoupled LoRA to decouple the attention architecture for spatial appearance and temporal motion processing. During the second training stage, we design the sparse motion sampling and adaptive RoPE to accelerate the tuning speed. To address the lack of a benchmark for this field, we introduce MotionBench, a comprehensive benchmark comprising diverse motion, including creative camera motion, single object motion, multiple object motion, and complex human motion. We show extensive evaluations on MotionBench to verify the superiority of Follow-Your-Motion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。