语言风格模型在量化下更抗损,性能优于扩散模型。
Dissecting Bit-Level Scaling Laws in Quantizing Vision Generative Models
- 对比语言与扩散模型的量化表现,发现前者更耐信息损失。
- 语言模型在各量化设置下均胜出,比特级扩展性更优。
- 提出新蒸馏方法提升量化后模型质量,适合部署优化者。
视觉生成模型近年来在扩散式和语言式两大范式上取得显著进展,二者均展现出良好扩展规律。量化对高效部署至关重要,可降低内存与计算开销。本文系统研究量化对两类模型的影响,发现尽管全精度下性能相当,语言式模型在各类量化设置中始终优于扩散式模型。这表明语言式模型具有更优的比特级扩展规律,能在模型质量与总比特数间实现更好权衡。深入分析发现,主要原因是语言式模型的离散表示空间对量化带来的信息损失更具容忍度。此外,我们指出改进量化模型的比特级扩展规律极具挑战性,而模型蒸馏被证明是高效手段。为此,我们提出TopKLD方法,在蒸馏过程中平衡‘隐含知识’与‘显式知识’,使整数与浮点量化下的比特级扩展规律整体提升一个等级。
原文摘要 · Abstract (English)
Vision generative models have recently made significant advancements along two primary paradigms: diffusion-style and language-style, both of which have demonstrated excellent scaling laws. Quantization is crucial for efficiently deploying these models, as it reduces memory and computation costs. In this work, we systematically investigate the impact of quantization on these two paradigms. Surprisingly, despite achieving comparable performance in full precision, language-style models consistently outperform diffusion-style models across various quantization settings. This observation suggests that language-style models have superior bit-level scaling laws, offering a better tradeoff between model quality and total bits. To dissect this phenomenon, we conduct extensive experiments and find that the primary reason is the discrete representation space of language-style models, which is more tolerant of information loss during quantization. Furthermore, our analysis indicates that improving the bit-level scaling law of quantized vision generative models is challenging, with model distillation identified as a highly effective approach. Specifically, we propose TopKLD to optimize the transfer of distilled knowledge by balancing ``implicit knowledge'' and ``explicit knowledge'' during the distillation process. This approach elevates the bit-level scaling laws by one level across both integer and floating-point quantization settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。