首个专为手机优化的视频扩散模型，实现高效生成。
Mobile Video Diffusion
- 通过降低分辨率、多尺度时序建模与通道/时序块剪枝，大幅减少计算开销。
- 仅需1.7秒生成14帧512×256分辨率视频，推理能耗降至4.34 TFLOPs（原模型1817.2）。
- 适合移动端部署，尤其适用于资源受限设备上的实时视频生成。
视频扩散模型虽已实现高保真与强可控性，但计算成本过高，难以在移动设备上运行。本文提出首个面向移动设备优化的视频扩散模型——MobileVD。基于Stable Video Diffusion（SVD）的时空UNet结构，通过降低帧分辨率、引入多尺度时序表示，并设计两种新型剪枝策略，有效减少通道数与时序模块数量。此外，采用对抗微调将去噪过程压缩至单步。实验表明，该模型效率提升523倍（从1817.2降至4.34 TFLOPs），FVD指标从171降至149，仅轻微损失画质，在Xiaomi-14 Pro上可于1.7秒内生成14×512×256像素的视频片段。相关结果详见https://qualcomm-ai-research.github.io/mobile-video-diffusion/
原文摘要 · Abstract (English)
Video diffusion models have achieved impressive realism and controllability but are limited by high computational demands, restricting their use on mobile devices. This paper introduces the first mobile-optimized video diffusion model. Starting from a spatio-temporal UNet from Stable Video Diffusion (SVD), we reduce memory and computational cost by reducing the frame resolution, incorporating multi-scale temporal representations, and introducing two novel pruning schema to reduce the number of channels and temporal blocks. Furthermore, we employ adversarial finetuning to reduce the denoising to a single step. Our model, coined as MobileVD, is 523x more efficient (1817.2 vs. 4.34 TFLOPs) with a slight quality drop (FVD 149 vs. 171), generating latents for a 14x512x256 px clip in 1.7 seconds on a Xiaomi-14 Pro. Our results are available at https://qualcomm-ai-research.github.io/mobile-video-diffusion/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。