让视频VAE在手机上快速稳定运行,首次实现720p实时解码。
Turbo-VAED: Fast and Stable Transfer of Video-VAEs to Mobile Devices
- 用3D深度可分离卷积减少参数,解决模型过大问题。
- 设计新型像素重组方案,使端到端延迟大幅降低。
- 仅蒸馏解码器即可适配手机,训练成本低且质量损失小。
随着大模型向移动端部署的需求增长,当前主流视频生成模型中的变分自编码器(VAE)成为主要计算瓶颈,因参数量大和核不匹配导致内存溢出或推理极慢。为此,本文提出低成本高效迁移方案,将通用视频VAE适配至移动设备。首先分析现有架构冗余,引入3D深度可分离卷积显著降低参数量;其次发现主流上采样方法不适应移动硬件,提出解耦式3D像素重组方案,大幅缩减延迟;最后提出仅蒸馏解码器的训练方法,使部署只需微调解码器,实现快速适配。所提通用移动端优化解码器Turbo-VAED,在4个代表性模型上验证:训练成本低至95美元,720p下加速最高达84.5倍,参数仅需原模型17.5%,重建质量保留96.9%。相比已有移动端优化方案,其在iPhone 16 Pro上提升2.9倍帧率并保持更优重建效果。代码与模型即将开源。
原文摘要 · Abstract (English)
There is a growing demand for deploying large generative AI models on mobile devices. For recent popular video generative models, however, the Variational AutoEncoder (VAE) represents one of the major computational bottlenecks. Both large parameter sizes and mismatched kernels cause out-of-memory errors or extremely slow inference on mobile devices. To address this, we propose a low-cost solution that efficiently transfers widely used video VAEs to mobile devices. (1) We analyze redundancy in existing VAE architectures and get empirical design insights. By integrating 3D depthwise separable convolutions into our model, we significantly reduce the number of parameters. (2) We observe that the upsampling techniques in mainstream video VAEs are poorly suited to mobile hardware and form the main bottleneck. In response, we propose a decoupled 3D pixel shuffle scheme that slashes end-to-end delay. Building upon these, we develop a universal mobile-oriented VAE decoder, Turbo-VAED. (3) We propose an efficient VAE decoder training method. Since only the decoder is used during deployment, we distill it to Turbo-VAED instead of retraining the full VAE, enabling fast mobile adaptation with minimal performance loss. To our knowledge, our method enables real-time 720p video VAE decoding on mobile devices for the first time. This approach is widely applicable to most video VAEs. When integrated into four representative models, with training cost as low as $95, it accelerates original VAEs by up to 84.5x at 720p resolution on GPUs, uses as low as 17.5% of original parameter count, and retains 96.9% of the original reconstruction quality. Compared to mobile-optimized VAEs, Turbo-VAED achieves a 2.9x speedup in FPS and better reconstruction quality on the iPhone 16 Pro. The code and models will soon be available at https://github.com/hustvl/Turbo-VAED.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。