提出新型稀疏注意力机制，实现分钟级长视频端到端生成
MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generation
- 用可学习的令牌路由机制精准匹配关键查询-键对，避免粗粒度分块
- 支持58万上下文长度，端到端生成24帧/秒的480p多镜头视频
- 无需核函数，兼容FlashAttention和序列并行，适合大规模视频生成
基于扩散Transformer（DiTs）的长视频生成受限于注意力机制随序列长度呈二次增长。由于注意力高度冗余，输出主要由少数查询-键对决定。现有稀疏方法依赖分块粗略估计，其精度与效率权衡受块大小制约。本文提出混合分组注意力（MoGA），一种高效的稀疏注意力机制，通过轻量级可学习的令牌路由器精确匹配令牌，无需分块估计。通过语义感知路由，MoGA实现了有效的长距离交互。作为无核方法，MoGA可无缝集成至现代注意力堆栈，包括FlashAttention和序列并行。基于MoGA，我们构建了一个高效长视频生成模型，可端到端生成分钟级、多镜头、480p、24 fps视频，上下文长度约58万。在多种视频生成任务上的实验验证了该方法的有效性。
原文摘要 · Abstract (English)
Long video generation with Diffusion Transformers (DiTs) is bottlenecked by the quadratic scaling of full attention with sequence length. Since attention is highly redundant, outputs are dominated by a small subset of query-key pairs. Existing sparse methods rely on blockwise coarse estimation, whose accuracy-efficiency trade-offs are constrained by block size. This paper introduces Mixture-of-Groups Attention (MoGA), an efficient sparse attention that uses a lightweight, learnable token router to precisely match tokens without blockwise estimation. Through semantic-aware routing, MoGA enables effective long-range interactions. As a kernel-free method, MoGA integrates seamlessly with modern attention stacks, including FlashAttention and sequence parallelism. Building on MoGA, we develop an efficient long video generation model that end-to-end produces minute-level, multi-shot, 480p videos at 24 fps, with a context length of approximately 580k. Comprehensive experiments on various video generation tasks validate the effectiveness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。