arXiv:2606.04945cs.LG2026-06

解决扩散模型量化中的状态与时间误差问题,提升低比特部署效率。

STaR-Quant: State-Time Consistent Post-Training Quantization for Diffusion Large Language Models

论文配图:STaR-Quant: State-Time Consistent Post-Training Quantization for Diffusion Large Language Models
图 1 · 摘自论文原文
  • 通过状态引导变换统一处理掩码与非掩码激活分布差异。
  • 引入时间注意力补偿机制,减轻迭代过程中的量化误差累积。
  • 在多个扩散模型上实现最高1.69倍加速与3.14倍内存节省。

扩散型大语言模型(DLLMs)通过双向上下文的迭代掩码去噪生成文本,是自回归模型的有力替代。然而其庞大模型规模和迭代去噪过程带来显著内存与计算开销,亟需后训练量化以实现高效部署。本文指出低比特量化面临两大挑战:状态相关的激活差异与时间上的误差累积——每步去噪中掩码与未掩码标记的激活分布不同,且量化误差在迭代过程中逐步积累。为此,我们提出STaR-Quant,一种面向扩散模型的状态-时间一致后训练量化框架。该方法引入状态引导激活变换(SGAT),通过统一的静态权重变换将掩码与未掩码标记分配至不同激活变换空间;进一步设计时间注意力补偿(TAC),采用轻量级块对角仿射映射修正量化后的注意力表示。在代表性DLLM上的实验表明,STaR-Quant在低比特权重-激活量化上持续优于强基线,相较FP16部署可实现最高1.69倍加速与3.14倍内存节省。

原文摘要 · Abstract (English)

Diffusion large language models (DLLMs) have recently emerged as a promising alternative to autoregressive LLMs by generating text through iterative masked denoising with bidirectional context. However, their large model sizes and iterative denoising process introduce substantial memory and computational overhead, motivating post-training quantization for efficient deployment. In this paper, we identify two key challenges for low-bit DLLM quantization: state-dependent activation disparity and temporal error accumulation. Masked and unmasked tokens exhibit different activation distributions within each denoising step, while quantization errors can accumulate across steps during iterative decoding. To address these challenges, we propose STaR-Quant, a state-time consistent PTQ framework for DLLMs. STaR-Quant introduces State-Guided Activation Transformation (SGAT) to assign masked and unmasked tokens to different activation transformation spaces with a unified static weight-side transformation. It further introduces Temporal Attention Compensation (TAC) to correct the quantized attention representation via a lightweight block-diagonal affine mapping. Experiments on representative DLLMs demonstrate that STaR-Quant consistently improves low-bit weight-activation quantization over strong PTQ baselines, while delivering up to 1.69x speedup and 3.14x memory saving over FP16 deployment.

扩散模型量化LLMPTQ

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。