arXiv:2411.01171cs.CVcs.AI2024-11NeurIPS被引 21

提出轻量推理框架,让消费级显卡也能生成高清视频

Fast and Memory-Efficient Video Diffusion Using Streamlined Inference

  • 通过分块特征与操作分组,降低内存占用
  • 利用扩散过程相似性跳过冗余步骤,加速推理
  • 在2080Ti上将内存从42GB降至11GB,适合普通用户

人工智能生成内容(AIGC)的快速发展,尤其是扩散模型的进步,显著推动了高质量视频生成技术的发展。然而,当前视频扩散模型对计算资源要求高,峰值内存消耗大,尤其在生成长时长、高分辨率视频时更为明显,严重制约其在标准硬件平台上的实际应用。为此,我们提出一种无需训练的新框架——Streamlined Inference,充分利用视频扩散模型在时空上的特性。该方法包含三个核心组件:Feature Slicer、Operator Grouping 和 Step Rehash。其中,Feature Slicer 将输入特征划分为子特征,Operator Grouping 对每个子特征使用一组连续算子处理,实现显著内存压缩而不损失质量或速度;Step Rehash 则利用扩散过程中相邻步骤的相似性,通过跳过不必要的步骤来加速推理。大量实验表明,该方法显著降低峰值内存与计算开销,使在单张消费级 GPU 上生成高质量视频成为可能(例如,将 AnimateDiff 的峰值内存从 42GB 降低至 11GB,且在 2080Ti 上实现更快推理)。

原文摘要 · Abstract (English)

The rapid progress in artificial intelligence-generated content (AIGC), especially with diffusion models, has significantly advanced development of high-quality video generation. However, current video diffusion models exhibit demanding computational requirements and high peak memory usage, especially for generating longer and higher-resolution videos. These limitations greatly hinder the practical application of video diffusion models on standard hardware platforms. To tackle this issue, we present a novel, training-free framework named Streamlined Inference, which leverages the temporal and spatial properties of video diffusion models. Our approach integrates three core components: Feature Slicer, Operator Grouping, and Step Rehash. Specifically, Feature Slicer effectively partitions input features into sub-features and Operator Grouping processes each sub-feature with a group of consecutive operators, resulting in significant memory reduction without sacrificing the quality or speed. Step Rehash further exploits the similarity between adjacent steps in diffusion, and accelerates inference through skipping unnecessary steps. Extensive experiments demonstrate that our approach significantly reduces peak memory and computational overhead, making it feasible to generate high-quality videos on a single consumer GPU (e.g., reducing peak memory of AnimateDiff from 42GB to 11GB, featuring faster inference on 2080Ti).

视频生成扩散模型内存优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。