针对扩散语言模型提出2比特极低精度量化方法，显著提升压缩后性能。
Quant-dLLM: Post-Training Extreme Low-Bit Quantization for Diffusion Large Language Models
- 通过模拟掩码校准匹配扩散模型的时序遮蔽特性，提升量化可靠性。
- 在2比特约束下，性能优于现有自回归模型迁移方法，保持高精度生成能力。
- 适合需要极致压缩且保留生成质量的部署场景，如边缘设备运行大模型。
扩散大语言模型（dLLMs）具备双向上下文和灵活的掩码去噪生成能力，正成为自回归（AR）大模型的有力替代。然而，其模型规模持续增长，亟需权重压缩以支持部署。尽管后训练量化（PTQ）对AR大模型有效，但直接将该方法应用于dLLMs时在2比特下表现不佳。为此，我们提出Quant-dLLM，一种专为dLLMs设计的超低比特PTQ框架。由于dLLMs中的掩码去噪激活与标准PTQ假设的全可见信号不同，我们引入掩码校准模拟（MCS），使其校准过程与随时间步变化的掩码机制对齐，从而获得更可靠的校准结果。此外，我们提出数据感知任意顺序量化器（DAQ），通过优化算法学习超低比特权重表示，并基于模拟校准数据进行迭代逼近。在严格2比特预算下，我们进一步引入基于敏感度的自适应块混合精度（ABMP）策略，动态分配通道组的位宽。在2比特约束下，Quant-dLLM在dLLMs上始终优于当前最优的AR迁移量化方法，实现更高准确率。代码与模型将公开于：https://github.com/ZTA2785/Quant-dLLM。
原文摘要 · Abstract (English)
Diffusion large language models (dLLMs), which offer bidirectional context and flexible masked-denoising generation, are emerging as a compelling alternative to autoregressive (AR) LLMs. However, like AR LLMs, their model sizes continue to grow, motivating weight compression for deployment. Although post-training quantization (PTQ) is effective for AR LLMs, directly transferring it to dLLMs at 2-bit leads to unsatisfactory performance. To tackle these challenges, we propose Quant-dLLM, an ultra-low-bit PTQ framework tailored to dLLMs. Since masked-denoising activations in dLLMs differ from the fully visible signals assumed by standard PTQ methods, we introduce Masked Calibration Simulation (MCS) to align calibration with the timestep-dependent masking, which yields more reliable calibrations. Moreover, we propose a Data-aware Any-order Quantizer (DAQ) that learns ultra-low-bit weight representations via an optimization algorithm. It performs iterative approximation guided by our simulated calibration data. In addition, under a strict 2-bit budget, we introduce Adaptive Blockwise Mixed Precision (ABMP), a sensitivity-based precision allocation scheme that adaptively assigns bit width across channel groups. When restricted to 2-bit precision, Quant-dLLM consistently achieves higher accuracy than state-of-the-art (SOTA) AR-transfer PTQ methods on dLLMs. The code and models will be available at: https://github.com/ZTA2785/Quant-dLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。