arXiv:2503.18559cs.CV2025-03被引 7

轻量级文生视频模型,效率提升31倍且支持26帧长视频生成

AMD-Hummingbird: Towards an Efficient Text-to-Video Model

  • 通过剪枝与视觉反馈学习,将U-Net参数减半至0.7亿
  • 在4块GPU上训练,比VideoCrafter2快31倍,VBench得分最高
  • 支持长视频生成与风格定制,适合移动端部署

文生视频(T2V)生成因能从文本描述合成真实视频而备受关注。然而,现有模型难以在计算效率与高画质之间取得平衡,尤其在资源受限设备如iGPUs和手机上表现不佳。多数先前工作侧重画质,忽视了适用于实际部署的小型高效模型需求。为此,我们提出轻量级T2V框架Hummingbird,通过剪枝现有模型并结合视觉反馈学习提升画质。该方法将U-Net参数从14亿降至7亿,显著提升效率,同时保持高质量视频生成能力。此外,我们引入基于大语言模型(LLMs)和视频质量评估(VQA)模型的新数据处理流程,优化文本提示与视频数据质量。为支持用户驱动训练与风格定制,我们公开完整训练代码(含数据处理与模型训练)。大量实验表明,本方法相比VideoCrafter2实现31倍加速,且在VBench上获得最高综合评分;支持生成最多26帧的长视频,突破了传统U-Net方法在长视频生成上的限制。值得注意的是,整个训练仅需4块GPU,性能可媲美当前领先方法。Hummingbird为T2V生成提供了兼具高性能、可扩展性与灵活性的实际解决方案。

原文摘要 · Abstract (English)

Text-to-Video (T2V) generation has attracted significant attention for its ability to synthesize realistic videos from textual descriptions. However, existing models struggle to balance computational efficiency and high visual quality, particularly on resource-limited devices, e.g.,iGPUs and mobile phones. Most prior work prioritizes visual fidelity while overlooking the need for smaller, more efficient models suitable for real-world deployment. To address this challenge, we propose a lightweight T2V framework, termed Hummingbird, which prunes existing models and enhances visual quality through visual feedback learning. Our approach reduces the size of the U-Net from 1.4 billion to 0.7 billion parameters, significantly improving efficiency while preserving high-quality video generation. Additionally, we introduce a novel data processing pipeline that leverages Large Language Models (LLMs) and Video Quality Assessment (VQA) models to enhance the quality of both text prompts and video data. To support user-driven training and style customization, we publicly release the full training code, including data processing and model training. Extensive experiments show that our method achieves a 31X speedup compared to state-of-the-art models such as VideoCrafter2, while also attaining the highest overall score on VBench. Moreover, our method supports the generation of videos with up to 26 frames, addressing the limitations of existing U-Net-based methods in long video generation. Notably, the entire training process requires only four GPUs, yet delivers performance competitive with existing leading methods. Hummingbird presents a practical and efficient solution for T2V generation, combining high performance, scalability, and flexibility for real-world applications.

文生视频轻量模型高效生成长视频

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。