针对扩散模型语言生成的量化难题,提出专有压缩方法。
DLLMQuant: Quantizing Diffusion-based Large Language Models
- 设计时序-掩码自适应采样,捕捉不同生成阶段的分布差异。
- 在W4A4下将准确率下降控制在16%以内,显著优于现有方法。
- 适合需要高效部署扩散式大模型的研究者与工程师。
基于扩散的大型语言模型(DLLMs)在非自回归文本生成方面展现出潜力,但其部署受限于模型规模庞大和计算开销高。后训练量化(PTQ)虽广泛用于压缩和加速大语言模型,但直接应用于DLLMs时会引发严重精度下降和泛化性能衰退(如AWQ在LLADA数据集上使用W4A4时准确率下降16%)。本文揭示了DLLMs的关键机制——动态掩码、迭代生成、双向注意力——与量化之间的根本冲突。发现三大核心问题:1)迭代生成与动态掩码比例导致解码各阶段的词分布差异显著,现有校准方法无法充分捕捉;2)量化误差在迭代过程中累积并放大,导致模型性能随解码步数增加而恶化;3)未掩码词趋于稳定,掩码词保持概率分布,整体特征分布与现有量化方法不兼容。为此,提出专为DLLMs设计的PTQ框架DLLMQuant,包含三项创新技术:1)时序-掩码自适应采样(TMAS),结合时间与掩码因素进行校准,可捕获跨时间步的分布;2)交互感知激活量化(IA-AQ),利用双向注意力的交互信号动态分配量化资源;3)确定性引导量化(CGQ),将掩码状态与词分值作为加权依据,优化误差补偿,使权重量化更适配DLLMs。实验表明,DLLMQuant在提升效率的同时实现显著性能提升。
原文摘要 · Abstract (English)
Diffusion-based large language models (DLLMs) have shown promise for non-autoregressive text generation, but their deployment is constrained by large model sizes and heavy computational costs. Post-training quantization (PTQ), a widely used method for compressing and accelerating Large Language Models (LLMs), suffers from severe accuracy degradation and reduced generalization performance when directly applied to DLLMs (e.g., AWQ suffers a 16% accuracy drop on LLADA under W4A4). This paper explores how DLLMs' key mechanisms - dynamic masking, iterative generation, bidirectional attention - clash with quantization. We identify three core issues: 1) Iterative generation and dynamic masking ratios lead to distinct token distributions across decoding steps, which are not adequately captured by existing PTQ calibration methods; 2) Quantization errors are accumulated and amplified progressively during iteration in DLLMs, causing quantized models to perform worse as decoding steps progress; 3) Unmasked tokens stabilize while masked remain probabilistic, making overall feature distribution incompatible with existing PTQ methods. To address these issues, we propose DLLMQuant, a PTQ framework tailored for DLLMs, which incorporates three novel techniques: 1) Temporal-Mask Adaptive Sampling (TMAS), a calibration method that accounts for both time and mask factors, with the capacity to capture distributions across timesteps. 2) Interaction-Aware Activation Quantization (IA-AQ), which utilizes bidirectional attention's interaction signals to dynamically allocate quantization resources. 3) Certainty-Guided Quantization (CGQ), which integrates mask status and token scores as key weighting criteria into error compensation, making weight quantization more suitable for DLLMs. Experiments show that DLLMQuant achieves significant performance gains while enhancing efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。