arXiv:2412.09856cs.CVcs.AI2024-12CVPR被引 47

用线性复杂度实现单卡高清分钟级视频生成,速度提升15倍。

LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexity

论文配图:LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexity
图 1 · 摘自论文原文
  • 用MATE模块替代自注意力,实现像素数线性增长的计算开销
  • 单卡生成1分钟高清视频,推理耗时降低11.5倍,显存占用大幅减少
  • 适合需要长视频实时生成的场景,如互动视频、电影级内容创作

文本到视频生成虽能提升内容创作效率,但计算成本极高:扩散变换器(DiTs)的计算量随像素数呈二次方增长,导致现有模型大多仅支持10-20秒视频生成。本文提出线性复杂度文本到视频生成框架LinGen,其计算成本随像素数线性增长。首次实现在单张GPU上生成高分辨率分钟级视频而不损失质量。该框架将计算主导的二次复杂度自注意力模块替换为线性复杂度的MATE模块,包含MA分支与TE分支。MA分支通过双向Mamba2块与旋转主扫描(Rotary Major Scan)及回顾令牌机制捕捉短至长距离相关性;TE分支为新型时间Swin注意力块,专注相邻与中等范围时序相关性。MATE模块解决了Mamba的邻接保持问题,显著提升生成视频的一致性。实验表明,LinGen在视频质量上超越DiT(胜率75.6%),计算量和延迟分别降低15×(11.5×)。自动评估与人工评价均显示,LinGen-4B在质量上可媲美顶尖模型(相较于Gen-3、LumaLabs、Kling的胜率分别为50.5%、52.1%、49.1%)。这为小时级电影生成与实时交互式视频生成铺平道路。项目网站提供68秒生成示例及其他案例:https://lineargen.github.io/

原文摘要 · Abstract (English)

Text-to-video generation enhances content creation but is highly computationally intensive: The computational cost of Diffusion Transformers (DiTs) scales quadratically in the number of pixels. This makes minute-length video generation extremely expensive, limiting most existing models to generating videos of only 10-20 seconds length. We propose a Linear-complexity text-to-video Generation (LinGen) framework whose cost scales linearly in the number of pixels. For the first time, LinGen enables high-resolution minute-length video generation on a single GPU without compromising quality. It replaces the computationally-dominant and quadratic-complexity block, self-attention, with a linear-complexity block called MATE, which consists of an MA-branch and a TE-branch. The MA-branch targets short-to-long-range correlations, combining a bidirectional Mamba2 block with our token rearrangement method, Rotary Major Scan, and our review tokens developed for long video generation. The TE-branch is a novel TEmporal Swin Attention block that focuses on temporal correlations between adjacent tokens and medium-range tokens. The MATE block addresses the adjacency preservation issue of Mamba and improves the consistency of generated videos significantly. Experimental results show that LinGen outperforms DiT (with a 75.6% win rate) in video quality with up to 15$\times$ (11.5$\times$) FLOPs (latency) reduction. Furthermore, both automatic metrics and human evaluation demonstrate our LinGen-4B yields comparable video quality to state-of-the-art models (with a 50.5%, 52.1%, 49.1% win rate with respect to Gen-3, LumaLabs, and Kling, respectively). This paves the way to hour-length movie generation and real-time interactive video generation. We provide 68s video generation results and more examples in our project website: https://lineargen.github.io/.

视频生成扩散模型线性复杂度长视频

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。