提出浮点量化方法，让扩散Transformer模型在4位权重下仍保持高质量图像生成。
FP4DiT: Towards Effective Floating Point Quantization for Diffusion Transformers
- 采用浮点量化技术，改进权重校准与激活量化机制。
- 在W4A6和W4A8精度下，图像质量优于传统整数量化方法。
- 适用于PixArt、Hunyuan等新型扩散变压器模型，适合边缘部署场景。
扩散模型（DM）革新了文本到图像的生成过程，但其高计算开销和模型体积限制了在边缘设备上的部署。后训练量化（PTQ）是一种无需训练或微调的轻量级解决方案。尽管现有方法在基于卷积的DM上实现了W4A8（4位权重，8位激活），但在采用新型扩散变压器（DiT）架构的模型如PixArt系列、Hunyuan中仍存在局限：当前多数研究聚焦于旧式卷积网络，而对新式变压器结构关注不足；且主流整数量化（INT）难以匹配权重与激活分布，浮点量化（FPQ）潜力未被充分挖掘。本文提出FP4DiT，一种基于浮点量化的PTQ方法，实现W4A6量化。通过扩展自适应舍入技术优化权重量化，并揭示DiT激活依赖输入图像块数据，需在线激活量化机制。实验表明，FP4DiT在W4A6与W4A8精度下，相较整数量化，在CLIP、ImageReward及HPSv2指标上表现更优，且在PixArt-α、PixArt-Σ与Hunyuan上生成高质量图像。
原文摘要 · Abstract (English)
Diffusion Models (DM) have revolutionized the text-to-image visual generation process. However, the large computational cost and model footprint of DMs hinders practical deployment, especially on edge devices. Post-training quantization (PTQ) is a lightweight method to alleviate these burdens without the need for training or fine-tuning. While recent DM PTQ methods achieve W4A8 \blue{(i.e., 4-bit weights and 8-bit activations)} on integer-based PTQ, two key limitations remain: First, while most existing DM PTQ methods evaluate on classical DMs like Stable Diffusion XL, 1.5 or earlier, which use convolutional U-Nets, newer Diffusion Transformer (DiT) models like the PixArt series, Hunyuan and others adopt fundamentally different transformer backbones to achieve superior image synthesis. Second, integer (INT) quantization is prevailing in DM PTQ but does not align well with the network weight and activation distribution, while Floating-Point Quantization (FPQ) is still under-investigated, yet it holds the potential to better align the weight and activation distributions in low-bit settings for DiT. In this paper, we introduce FP4DiT, a PTQ method that leverages FPQ to achieve W4A6 quantization. Specifically, we extend and generalize the Adaptive Rounding PTQ technique to adequately calibrate weight quantization for FPQ and demonstrate that DiT activations depend on input patch data, necessitating robust online activation quantization techniques. Experimental results demonstrate that FP4DiT achieves higher CLIP, ImageReward and HPSv2 performance compared to integer-based PTQ at the W4A6 and W4A8 precision levels while generating convincing visual content on PixArt-$α$, PixArt-$Σ$ and Hunyuan.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。