提出矩阵注意力机制，让视频生成更高效且连贯。
FrameDiT: Diffusion Transformer with Matrix Attention for Efficient Video Generation
- 用矩阵操作实现帧级时序注意力，捕捉全局时空结构。
- 在多个基准上达到最好效果，质量和连贯性双提升。
- 适合需要高效高质量视频生成的研究与应用。
高保真视频生成对扩散模型仍是挑战，主要源于难以高效建模复杂的时空动态。现有视频扩散方法通常将视频表示为时空标记序列，使用扩散Transformer（DiT）建模。但此类方法在强而昂贵的全3D注意力与高效但时序受限的局部分解注意力之间存在权衡。为此，我们提出矩阵注意力（Matrix Attention），一种帧级时序注意力机制，将整帧作为矩阵处理，通过原生矩阵运算生成查询、键和值矩阵。通过跨帧而非标记进行注意力计算，该机制有效保持全局时空结构并适应显著运动。我们构建了基于矩阵注意力的FrameDiT-G，并进一步推出FrameDiT-H，融合矩阵注意力与局部分解注意力，以同时捕捉大尺度与小尺度运动。大量实验表明，FrameDiT-H在多个视频生成基准上取得当前最佳性能，在保持与局部分解注意力相当效率的同时，显著提升时间连贯性和视频质量。
原文摘要 · Abstract (English)
High-fidelity video generation remains challenging for diffusion models due to the difficulty of modeling complex spatio-temporal dynamics efficiently. Recent video diffusion methods typically represent a video as a sequence of spatio-temporal tokens which can be modeled using Diffusion Transformers (DiTs). However, this approach faces a trade-off between the strong but expensive Full 3D Attention and the efficient but temporally limited Local Factorized Attention. To resolve this trade-off, we propose Matrix Attention, a frame-level temporal attention mechanism that processes an entire frame as a matrix and generates query, key, and value matrices via matrix-native operations. By attending across frames rather than tokens, Matrix Attention effectively preserves global spatio-temporal structure and adapts to significant motion. We build FrameDiT-G, a DiT architecture based on MatrixAttention, and further introduce FrameDiT-H, which integrates Matrix Attention with Local Factorized Attention to capture both large and small motion. Extensive experiments show that FrameDiT-H achieves state-of-the-art results across multiple video generation benchmarks, offering improved temporal coherence and video quality while maintaining efficiency comparable to Local Factorized Attention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。