提出DGQ方法,让文生图模型低比特量化后仍保真且对齐文本。
DGQ: Distribution-Aware Group Quantization for Text-to-Image Diffusion Models
- 按分布识别像素和通道异常值,自适应处理以保图像质量
- 针对提示词使用对数量化尺度,维持文本与图像一致
- 首次实现无需微调的低比特文生图模型量化,适合部署场景
尽管文生图扩散模型应用广泛,但其计算与内存开销限制了实际应用。为缓解此问题,已有研究探索模型量化,通过将权重和激活压缩至低位格式来降低内存占用和计算成本。然而,现有方法在低位(<8位)量化时难以同时保持图像质量和文本-图像对齐性。本文从分布视角分析文生图模型量化挑战,发现激活异常值对图像质量影响显著,且跨注意力分数存在独特模式,影响文本-图像对齐。为此,我们提出分布感知分组量化(DGQ),通过识别并自适应处理像素级与通道级异常值,提升图像质量;同时采用提示词相关的对数量化尺度,维持文本-图像一致性。在MS-COCO和PartiPrompts数据集上表现优异。首次实现无需额外微调权重量化参数的低比特文生图模型量化。代码已开源。
原文摘要 · Abstract (English)
Despite the widespread use of text-to-image diffusion models across various tasks, their computational and memory demands limit practical applications. To mitigate this issue, quantization of diffusion models has been explored. It reduces memory usage and computational costs by compressing weights and activations into lower-bit formats. However, existing methods often struggle to preserve both image quality and text-image alignment, particularly in lower-bit($<$ 8bits) quantization. In this paper, we analyze the challenges associated with quantizing text-to-image diffusion models from a distributional perspective. Our analysis reveals that activation outliers play a crucial role in determining image quality. Additionally, we identify distinctive patterns in cross-attention scores, which significantly affects text-image alignment. To address these challenges, we propose Distribution-aware Group Quantization (DGQ), a method that identifies and adaptively handles pixel-wise and channel-wise outliers to preserve image quality. Furthermore, DGQ applies prompt-specific logarithmic quantization scales to maintain text-image alignment. Our method demonstrates remarkable performance on datasets such as MS-COCO and PartiPrompts. We are the first to successfully achieve low-bit quantization of text-to-image diffusion models without requiring additional fine-tuning of weight quantization parameters. Code is available at https://github.com/ugonfor/DGQ.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。