Neodragon在手机上6.7秒生成49帧高清视频，实现端侧高效视频生成。
Neodragon: Mobile Video Generation using Diffusion Transformer
- 用小模型替代大模型，通过蒸馏技术保持画质
- 4.945B参数模型在手机上仅用3.5GB内存，6.7秒完成生成
- 适合移动设备开发者与关注隐私的创作者使用
我们提出Neodragon，一个可在高通Hexagon NPU上直接生成2秒（49帧@24fps）1024x640分辨率视频的文本到视频系统，创下6.7秒（7FPS）的记录。不同于传统基于Transformer的离线生成模型，Neodragon是首个专为移动硬件优化的高效高保真视频合成系统。通过四项关键技术：(1) 以0.2B的DT5替代原4.762B T5xxl文本编码器，通过新型文本编码器蒸馏实现质量损失极小；(2) 提出非对称解码器蒸馏，替换原有编解码-潜在VAE解码器，不破坏生成潜空间；(3) 基于重要性剪枝去噪器中的MMDiT模块，并通过两阶段蒸馏恢复性能；(4) 采用适配金字塔流匹配的DMD进行步骤蒸馏，显著降低去噪器的NFE需求，大幅加速生成。结合优化的SSD1B首帧图像生成器和QuickSRNet超分网络，整个系统达到4.945B总参数、峰值3.5GB内存、端到端6.7秒延迟，同时获得VBench总分81.61。该系统实现了低成本、私密且端侧的视频生成，推动AI视频创作民主化。
原文摘要 · Abstract (English)
We introduce Neodragon, a text-to-video system capable of generating 2s (49 frames @24 fps) videos at the 640x1024 resolution directly on a Qualcomm Hexagon NPU in a record 6.7s (7 FPS). Differing from existing transformer-based offline text-to-video generation models, Neodragon is the first to have been specifically optimised for mobile hardware to achieve efficient and high-fidelity video synthesis. We achieve this through four key technical contributions: (1) Replacing the original large 4.762B T5xxl Text-Encoder with a much smaller 0.2B DT5 (DistilT5) with minimal quality loss, enabled through a novel Text-Encoder Distillation procedure. (2) Proposing an Asymmetric Decoder Distillation approach allowing us to replace the native codec-latent-VAE decoder with a more efficient one, without disturbing the generative latent-space of the generation pipeline. (3) Pruning of MMDiT blocks within the denoiser backbone based on their relative importance, with recovery of original performance through a two-stage distillation process. (4) Reducing the NFE (Neural Functional Evaluation) requirement of the denoiser by performing step distillation using DMD adapted for pyramidal flow-matching, thereby substantially accelerating video generation. When paired with an optimised SSD1B first-frame image generator and QuickSRNet for 2x super-resolution, our end-to-end Neodragon system becomes a highly parameter (4.945B full model), memory (3.5GB peak RAM usage), and runtime (6.7s E2E latency) efficient mobile-friendly model, while achieving a VBench total score of 81.61. By enabling low-cost, private, and on-device text-to-video synthesis, Neodragon democratizes AI-based video content creation, empowering creators to generate high-quality videos without reliance on cloud services. Code and model will be made publicly available at our website: https://qualcomm-ai-research.github.io/neodragon
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。