arXiv:2508.14896cs.CLcs.AI2025-08被引 27

首次系统研究扩散型大模型的后训练量化,发现激活值异常是关键瓶颈。

Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs

  • 识别出扩散模型中异常大的激活值,影响低比特量化精度
  • 在多任务、多模型上验证主流量化方法,揭示不同配置下的性能差异
  • 为边缘设备部署扩散语言模型提供实用量化方案,适合模型压缩研究者

扩散大语言模型(dLLMs)为自然语言生成提供了不同于自回归模型的新路径,采用全注意力机制和基于去噪的解码策略。然而,由于参数量巨大、资源消耗高,其在边缘设备上的部署仍面临挑战。尽管后训练量化(PTQ)已被广泛用于压缩自回归大模型,但其在dLLMs中的适用性尚未被充分探索。本文首次系统研究了扩散语言模型的后训练量化问题。我们发现激活值存在异常值,表现为远超正常范围的极大数值,显著拉宽动态范围,成为低比特量化的主要障碍。更重要的是,我们实现了多种先进PTQ方法,并在多个任务类型与模型变体上进行了全面评估。分析围绕比特位宽、量化方法、任务类别和模型类型四个维度展开。通过多角度实验,揭示了dLLMs在不同配置下的量化行为规律,为未来高效部署扩散模型提供了实践依据。代码已开源:https://github.com/FelixMessi/QDLM。

原文摘要 · Abstract (English)

Recent advances in diffusion large language models (dLLMs) have introduced a promising alternative to autoregressive (AR) LLMs for natural language generation tasks, leveraging full attention and denoising-based decoding strategies. However, the deployment of these models on edge devices remains challenging due to their massive parameter scale and high resource demands. While post-training quantization (PTQ) has emerged as a widely adopted technique for compressing AR LLMs, its applicability to dLLMs remains largely unexplored. In this work, we present the first systematic study on quantizing diffusion-based language models. We begin by identifying the presence of activation outliers, characterized by abnormally large activation values that dominate the dynamic range. These outliers pose a key challenge to low-bit quantization, as they make it difficult to preserve precision for the majority of values. More importantly, we implement state-of-the-art PTQ methods and conduct a comprehensive evaluation across multiple task types and model variants. Our analysis is structured along four key dimensions: bit-width, quantization method, task category, and model type. Through this multi-perspective evaluation, we offer practical insights into the quantization behavior of dLLMs under different configurations. We hope our findings provide a foundation for future research in efficient dLLM deployment. Our code is publicly available at https://github.com/FelixMessi/QDLM.

扩散模型量化大模型压缩边缘部署

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。