用多项式混合器替代注意力机制，实现高效图像视频生成。
PoM: Efficient Image and Video Generation with the Polynomial Mixer
- 提出多项式混合器（PoM），线性复杂度替代二次注意力
- 在保持高质量生成的同时，内存与计算量显著降低
- 适合需要高效推理的图像视频生成场景
基于多头注意力（MHA）的扩散模型已成为生成高质量图像和视频的主流方法。然而，将图像或视频编码为补丁序列会导致注意力模式开销巨大，内存与计算需求随补丁数呈平方增长。为此，我们提出一种可直接替换MHA的多项式混合器（PoM），能将整个序列编码为显式状态，具有与令牌数线性相关的复杂度。该显式状态支持逐帧生成，极大降低内存与计算消耗，同时仍可并行训练。我们证明了PoM是通用序列到序列逼近器，与标准MHA等价。将PoM应用于多个扩散Transformer（DiT）模型，在图像与视频生成任务中均实现了高质量结果，且资源消耗更低。代码已公开于https://github.com/davidpicard/HoMM。
原文摘要 · Abstract (English)
Diffusion models based on Multi-Head Attention (MHA) have become ubiquitous to generate high quality images and videos. However, encoding an image or a video as a sequence of patches results in costly attention patterns, as the requirements both in terms of memory and compute grow quadratically. To alleviate this problem, we propose a drop-in replacement for MHA called the Polynomial Mixer (PoM) that has the benefit of encoding the entire sequence into an explicit state. PoM has a linear complexity with respect to the number of tokens. This explicit state also allows us to generate frames in a sequential fashion, minimizing memory and compute requirement, while still being able to train in parallel. We show the Polynomial Mixer is a universal sequence-to-sequence approximator, just like regular MHA. We adapt several Diffusion Transformers (DiT) for generating images and videos with PoM replacing MHA, and we obtain high quality samples while using less computational resources. The code is available at https://github.com/davidpicard/HoMM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。