将预训练视频模型转为分层结构，高效推理不损失画质。
PyramidalWan: On Making Pretrained Video Model Pyramidal for Efficient Inference
- 用低成本微调将预训练模型转为分层推理架构。
- 在多步去噪中降低计算量，输出视频质量无下降。
- 对比多种步骤蒸馏策略，提升推理效率适合部署场景。
近期提出的分层模型将传统的前后向扩散过程分解为多个在不同分辨率下运行的阶段。这些模型在低分辨率处理高噪声输入，而在高分辨率处理低噪声输入。这种分层方法显著降低了多步去噪模型的推理计算成本。然而，现有的开源分层视频模型均从头训练，视觉合理性方面相比最先进系统表现较差。本文提出一种流水线，通过低成本微调将预训练扩散模型转化为分层模型，实现转换过程且输出视频质量无下降。此外，我们研究并比较了分层模型中不同步骤蒸馏策略，以进一步提升推理效率。相关结果见 https://qualcomm-ai-research.github.io/PyramidalWan。
原文摘要 · Abstract (English)
Recently proposed pyramidal models decompose the conventional forward and backward diffusion processes into multiple stages operating at varying resolutions. These models handle inputs with higher noise levels at lower resolutions, while less noisy inputs are processed at higher resolutions. This hierarchical approach significantly reduces the computational cost of inference in multi-step denoising models. However, existing open-source pyramidal video models have been trained from scratch and tend to underperform compared to state-of-the-art systems in terms of visual plausibility. In this work, we present a pipeline that converts a pretrained diffusion model into a pyramidal one through low-cost finetuning, achieving this transformation without degradation in quality of output videos. Furthermore, we investigate and compare various strategies for step distillation within pyramidal models, aiming to further enhance the inference efficiency. Our results are available at https://qualcomm-ai-research.github.io/PyramidalWan.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。