针对视频生成中4位量化注意力的两大难题,提出旋转感知与范围优化新方法。
RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation

- 设计旋转感知机制,解决3D RoPE与量化旋转矩阵的冲突问题
- 通过固定缩放因子实现INT4全动态范围利用,提升量化精度
- 适用于需高速推理的DiT视频生成模型,显著加速且保持画质
基于DiT的视频生成模型采用3D旋转位置编码(3D RoPE)时,注意力机制因序列长度的平方复杂度仍是主要计算瓶颈。尽管量化FlashAttention为硬件加速提供可行路径,但现有低比特量化方法忽略两个关键挑战:一是在线旋转矩阵难以与RoPE兼容;二是非负注意力矩阵P = exp(QK - max(QK))使对称量化浪费一半4位动态范围。本文发现,Q和K的异常值分布受3D RoPE维度划分影响显著。据此提出RotateAttention,一种专用于带3D RoPE的DiT视频生成模型的混合精度INT4 FlashAttention框架,对敏感注意力块和去噪步骤采用选择性FP16回退。核心技术包括:1)RoPE感知旋转,使用可融合进RoPE或开销极小的旋转矩阵缓解RoPE引入的异常值;2)范围优化的P量化,通过固定缩放因子和零点充分挖掘INT4数值范围,计算开销极低。实验表明,RotateAttention在几乎保持全精度基线画质的同时,实现高达1.68倍端到端加速和2.2倍内核级加速。
原文摘要 · Abstract (English)
In $\textbf{DiT-based video generation models equipped with 3D Rotary Position Embeddings (3D RoPE)}$, the attention mechanism remains a primary computational bottleneck due to its quadratic complexity with respect to sequence length. While quantized $\textbf{FlashAttention}$ offers a promising path toward hardware acceleration, existing low-bit quantization methods overlook two critical challenges in this setting: $\textbf{1)}$ applying online rotation matrices -- a widely used technique for mitigating outliers in Queries ($Q$) and Keys ($K$) -- is difficult to reconcile with $\textbf{RoPE}$; and $\textbf{2)}$ the non-negative attention matrix $P = \exp(QK - \max(QK))$ makes symmetric quantization waste half of the 4-bit dynamic range. In this work, we observe that the outlier distributions of $Q$ and $K$ are strongly affected by the dimensional partitioning of $\textbf{3D RoPE}$. Based on this finding, we propose $\textbf{RotateAttention}$, an efficient $\textbf{mixed-precision INT4 FlashAttention}$ framework tailored for $\textbf{DiT-based video generation models with 3D RoPE}$, using selective $\textbf{FP16 fallback}$ for accuracy-sensitive attention blocks and denoising steps. RotateAttention introduces two core techniques: $\textbf{1) RoPE-aware Rotation}$, which employs either mergeable rotation matrices that can be fused into RoPE or negligible-overhead matrices to mitigate RoPE-induced outliers in $Q$ and $K$; and $\textbf{2) Range-optimized $P$ Quantization}$, which uses fixed scales and zero-points to fully exploit the $\textbf{INT4 numerical range}$ with minimal computational overhead. Experiments show that $\textbf{RotateAttention}$ preserves video generation quality nearly identical to full-precision baselines while achieving up to 1.68$\times$ end-to-end speedup and 2.2$\times$ kernel-level acceleration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。