针对视觉自回归模型的量化难题，提出新型量化方法提升精度。
Shift-and-Sum Quantization for Visual Autoregressive Models
- 通过值令牌对称移位求和降低注意力计算误差
- 用重采样校准数据使码本采样频率匹配预测概率
- 在图像生成与修复任务中实现量化新纪录
后训练量化（PTQ）可通过少量数据实现深度网络的高效部署。然而，其在视觉自回归模型（VAR）中的应用仍较少被研究。我们识别出两个关键挑战：(i) 在粗粒度尺度下，注意力-值乘积的重建误差较大，因高注意力得分更频繁出现；(ii) 由于校准数据有限，码本条目采样频率与其预测概率之间存在偏差。为此，我们提出专为VAR设计的PTQ框架。首先，引入一种移位-求和量化方法，通过对值令牌的对称移位副本进行量化结果聚合，减少重建误差；其次，提出一种校准数据重采样策略，使码本条目的采样频率与预测概率一致。在类别条件图像生成、图像修复、外推及编辑任务上的实验表明，该方法在多种VAR架构上均实现一致提升，建立了VAR领域PTQ的新基准。
原文摘要 · Abstract (English)
Post-training quantization (PTQ) enables efficient deployment of deep networks using a small set of data. Its application to visual autoregressive models (VAR), however, remains relatively unexplored. We identify two key challenges for applying PTQ to VAR: (i) large reconstruction errors in attention-value products, especially at coarse scales where high attention scores occur more frequently; and (ii) a discrepancy between the sampling frequencies of codebook entries and their predicted probabilities due to limited calibration data. To address these challenges, we propose a PTQ framework tailored for VAR. First, we introduce a shift-and-sum quantization method that reduces reconstruction errors by aggregating quantized results from symmetrically shifted duplicates of value tokens. Second, we present a resampling strategy for calibration data that aligns sampling frequencies of codebook entries with their predicted probabilities. Experiments on class-conditional image generation, inpainting, outpainting, and class-conditional editing show consistent improvements across VAR architectures, establishing a new state of the art in PTQ for VAR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。