用4步生成高质量人物视频,速度提升10倍且表情更自然
MagicDistillation: Weak-to-Strong Video Distillation for Large-Scale Few-Step Synthesis
- 用弱数据蒸馏强模型,通过分布匹配优化生成质量
- 4步合成视频,比原模型快10倍,FID/FVD指标更优
- 适合需要快速生成真实人脸动作的视频应用
近年来,开源视频扩散模型(如WanX、Magic141和HunyuanVideo)已扩展至超过100亿参数。这些大规模模型在视觉质量与运动自然性方面显著优于小规模模型,但仍面临两大挑战:(1) 推理开销高:在单张H100 GPU上生成28步视频需约10分钟;(2) 人物视频生成受限:如WanX-I2V和HunyuanVideo-I2V常生成不自然的面部表情与动作。为此,我们提出MagicDistillation框架,旨在降低推理开销并提升人物视频生成的泛化能力。具体而言,我们使用高质量对话视频微调Magic141以专注人物视频生成,并在分布匹配蒸馏(DMD)框架下,采用LoRA高效微调伪DiT。随后,通过弱到强(W2S)分布匹配,最小化伪数据分布与真实分布的差异,从而提升生成视频的视觉保真度与运动动态。在人物视频合成任务上的实验表明,MagicDistillation在FID/FVD指标及VBench评测中均优于Euler、LCM和DMD基线。此外,仅需4步即可超越WanX-I2V(14B)和HunyuanVideo-I2V(13B)在可视化效果与VBench表现。项目主页:https://magicdistillation.github.io/MagicDistillation/
原文摘要 · Abstract (English)
Recently, open-source video diffusion models (VDMs), such as WanX, Magic141 and HunyuanVideo, have been scaled to over 10 billion parameters. These large-scale VDMs have demonstrated significant improvements over smaller-scale VDMs across multiple dimensions, including enhanced visual quality and more natural motion dynamics. However, these models face two major limitations: (1) High inference overhead: Large-scale VDMs require approximately 10 minutes to synthesize a 28-step video on a single H100 GPU. (2) Limited in portrait video synthesis: Models like WanX-I2V and HunyuanVideo-I2V often produce unnatural facial expressions and movements in portrait videos. To address these challenges, we propose MagicDistillation, a novel framework designed to reduce inference overhead while ensuring the generalization of VDMs for portrait video synthesis. Specifically, we primarily use sufficiently high-quality talking video to fine-tune Magic141, which is dedicated to portrait video synthesis. We then employ LoRA to effectively and efficiently fine-tune the fake DiT within the step distillation framework known as distribution matching distillation (DMD). Following this, we apply weak-to-strong (W2S) distribution matching and minimize the discrepancy between the fake data distribution and the ground truth distribution, thereby improving the visual fidelity and motion dynamics of the synthesized videos. Experimental results on portrait video synthesis demonstrate the effectiveness of MagicDistillation, as our method surpasses Euler, LCM, and DMD baselines in both FID/FVD metrics and VBench. Moreover, MagicDistillation, requiring only 4 steps, also outperforms WanX-I2V (14B) and HunyuanVideo-I2V (13B) on visualization and VBench. Our project page is https://magicdistillation.github.io/MagicDistillation/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。