让扩散Transformer在手机上秒级生成视频。
Taming Diffusion Transformer for Efficient Mobile Video Generation in Seconds
- 用压缩自编码器降维，保持画质同时减少计算量。
- 三阶段剪枝加对抗性步数蒸馏，推理速度达15帧/秒。
- 适合移动端高效视频生成，尤其适合资源受限设备。
扩散 Transformer（DiT）在视频生成任务中表现优异，但其高计算开销使其难以在资源受限的设备（如智能手机）上实用，设备端部署尤为困难。本文提出一系列创新优化，显著加速视频生成并实现移动端实际部署。首先，采用高度压缩的变分自编码器（VAE）降低输入数据维度，同时保持视觉质量。其次，引入基于知识蒸馏引导、敏感度感知的三阶段剪枝策略，将模型规模压缩至适配移动平台，同时保留关键性能特征。第三，设计针对 DiT 的对抗性步数蒸馏技术，将推理步数降至四步。综合优化后，模型在 iPhone 16 Pro Max 上实现约 15 帧每秒（FPS）的生成速度，证明了在移动设备上实现高效、高质量视频生成的可行性。
原文摘要 · Abstract (English)
Diffusion Transformers (DiT) have shown strong performance in video generation tasks, but their high computational cost makes them impractical for resource-constrained devices like smartphones, and practical on-device generation is even more challenging. In this work, we propose a series of novel optimizations to significantly accelerate video generation and enable practical deployment on mobile platforms. First, we employ a highly compressed variational autoencoder (VAE) to reduce the dimensionality of the input data without sacrificing visual quality. Second, we introduce a KD-guided, sensitivity-aware tri-level pruning strategy to shrink the model size to suit mobile platforms while preserving critical performance characteristics. Third, we develop an adversarial step distillation technique tailored for DiT, which allows us to reduce the number of inference steps to four. Combined, these optimizations enable our model to achieve approximately 15 frames per second (FPS) generation speed on an iPhone 16 Pro Max, demonstrating the feasibility of efficient, high-quality video generation on mobile devices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。