提出旋转量化方法ConvRot，实现扩散模型4比特推理无损加速。
ConvRot: Rotation-Based Plug-and-Play 4-bit Quantization for Diffusion Transformers
- 用分组旋转与哈达玛变换抑制行列异常值
- 实现无需重训练的W4A4推理，速度提升2.26倍、内存减少4.05倍
- 适合部署大模型图像生成系统，可即插即用
扩散变压器在生成高质量图像方面表现出强大能力。然而，随着模型规模增大，日益增长的内存占用和推理延迟给实际部署带来挑战。近期大型语言模型研究显示，基于旋转的方法能平滑异常值并支持4比特量化，但这些方法常伴随显著开销，且难以处理扩散变压器中的行级异常值。为此，我们提出ConvRot，一种基于分组旋转的量化方法，利用正则哈达玛变换（RHT）抑制行级与列级异常值，同时将复杂度从二次降至线性。在此基础上，设计了ConvLinear4bit模块，集成旋转、量化、GEMM与反量化功能，实现无需重训练的W4A4推理，保持视觉质量。在FLUX.1-dev上的实验表明，该方法带来2.26倍的速度提升和4.05倍的内存减少，同时维持图像保真度。据我们所知，这是首次将基于旋转的量化应用于扩散变压器的即插即用式W4A4推理。
原文摘要 · Abstract (English)
Diffusion transformers have demonstrated strong capabilities in generating high-quality images. However, as model size increases, the growing memory footprint and inference latency pose significant challenges for practical deployment. Recent studies in large language models (LLMs) show that rotation-based techniques can smooth outliers and enable 4-bit quantization, but these approaches often incur substantial overhead and struggle with row-wise outliers in diffusion transformers. To address these challenges, we propose ConvRot, a group-wise rotation-based quantization method that leverages regular Hadamard transform (RHT) to suppress both row-wise and column-wise outliers while reducing complexity from quadratic to linear. Building on this, we design ConvLinear4bit, a plug-and-play module that integrates rotation, quantization, GEMM, and dequantization, enabling W4A4 inference without retraining and preserving visual quality. Experiments on FLUX.1-dev demonstrate a 2.26$\times$ speedup and 4.05$\times$ memory reduction while maintaining image fidelity. To our knowledge, this is the first application of rotation-based quantization for plug-and-play W4A4 inference in diffusion transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。