通过分频注意力机制,显著降低长视频生成的计算与内存开销。
FreqFormer: Hierarchical Frequency-Domain Attention with Adaptive Spectral Routing for Long-Sequence Video Diffusion Transformers
- 按频率分 band 使用不同注意力策略:低频全局、中频稀疏、高频局部
- 动态路由机制根据去噪阶段分配计算资源,早期重全局结构,后期重细节
- 支持64K至100万帧的长视频生成,适合部署在真实硬件上
长序列视频扩散变换器面临自注意力计算量呈二次增长的问题,显著影响运行时与内存占用。现有高效注意力方法普遍采用统一近似,但视频特征具有频谱结构:低频蕴含整体布局与粗略运动,高频包含纹理与精细细节。本文提出FreqFormer,一种基于频谱感知的异构注意力框架。将令牌特征按频带分离,分别采用压缩后的低频稠密全局注意力、中频块稀疏注意力和高频滑动窗口局部注意力。轻量级频谱路由网络依据层统计信息与扩散时间步,动态分配注意力头,使计算重心随去噪过程从全局结构转向细节。跨频带摘要令牌实现低成本残差传递。配合融合式GPU执行方案,协同调度稠密、稀疏与局部分支,减少核启动次数与内存传输。提供一致的复杂度模型、近似性的正交分解视角,以及基于仿真系统级数据(吞吐量、算术强度、内存流量、时延扩展)。在64K至100万令牌的模拟中,相比稠密注意力,FreqFormer大幅降低估计的注意力浮点运算量与KV相关内存流量,同时保持硬件友好的计算模式,验证了基于频谱结构的异构注意力是长视频扩散变换器的实际可行方向。
原文摘要 · Abstract (English)
Long-sequence video diffusion transformers hit a quadratic self-attention cost that dominates runtime and memory for very long token sequences. Most efficient attention methods use one approximation everywhere, yet video features are spectrally structured: low frequencies carry global layout and coarse motion; high frequencies carry texture and fine detail. We present FreqFormer, a frequency-aware heterogeneous attention framework. Token features are split into spectral bands with different operators: dense global attention on compressed low-frequency content, structured block-sparse attention on mid frequencies, and sliding-window local attention on high frequencies. A lightweight spectral routing network allocates heads across bands using layer statistics and the diffusion timestep, shifting compute toward global structure early in denoising and detail later. Cross-band summary tokens provide cheap residual exchange. FreqFormer is paired with a fused GPU execution plan that co-schedules dense, sparse, and local branches to cut kernel launches and memory traffic. We give a consistent complexity model, an orthonormal-decomposition view of approximation, and simulation-based systems numbers (throughput, arithmetic intensity, memory traffic, duration scaling). In simulations from 64K to 1M tokens, FreqFormer substantially reduces estimated attention FLOPs and KV-related memory traffic versus dense attention while keeping a hardware-friendly pattern, supporting spectrally structured heterogeneous attention as a practical direction for long-video diffusion transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。