arXiv:2608.18147cs.LGcs.AI2026-08

提出新量化方法,在压缩数据时兼顾精度与编码效率。

Entropy-Constrained Adaptive Stochastic Quantization

论文配图:Entropy-Constrained Adaptive Stochastic Quantization
图 1 · 摘自论文原文
  • 联合优化量化值与熵约束,提升压缩后精度。
  • 最优解时间复杂度为O(sd²),GPU适配近似解更高效。
  • 适合需高压缩比且对精度敏感的模型/梯度压缩场景。

自适应随机量化(ASQ)是一种近期提出的量化方法,可在保持无偏性的前提下最小化给定输入的均方误差(MSE),用于缓解现代机器学习工作负载中的通信和内存瓶颈,包括模型、梯度和KV缓存压缩以及最近邻搜索。实际系统可进一步使用无损熵编码压缩量化数据。然而,现有无偏方法(包括ASQ)在选择量化值时未考虑后续编码阶段,导致精度损失。本文提出熵约束自适应随机量化(ECASQ)问题,联合选择自适应量化值以在熵预算和无偏性约束下最小化MSE。我们给出一个最优动态规划算法,时间复杂度为O(sd²),空间复杂度O(d²),适用于长度d的向量和最多s个量化值;同时提供一种GPU友好的近似动态规划,时间复杂度仍为O(sd²),空间复杂度降至O(d)。该近似保证其解的MSE不超过使用少1比特熵的最优解。此外,我们还设计了迭代精化流程,实验表明其在保持显著速度优势的同时接近最优性能。

原文摘要 · Abstract (English)

Adaptive stochastic quantization (ASQ) is a recently introduced quantization approach that optimizes the Mean Squared Error (MSE) for a given input while preserving unbiasedness. It is designed to alleviate the communication and memory bottlenecks of modern data and machine learning workloads, including model, gradient, and KV-cache compression and nearest-neighbor search. Further, practical systems can then compress quantized data with a lossless entropy encoder. However, existing unbiased methods, including ASQ, choose their quantization values without considering this later encoding stage, leaving accuracy on the table. We formulate the Entropy Constrained Adaptive Stochastic Quantization (ECASQ) problem, which jointly selects adaptive quantization values to minimize MSE under an entropy budget and an unbiasedness constraint. We give an optimal dynamic program with $O(sd^2)$ time and $O(d^2)$ space for a length-d vector and at most s quantization values, and a GPU-friendly approximate dynamic program with $O(sd^2)$ time and $O(d)$ space. The approximation guarantees that the solution has an MSE no larger than the optimal solution that uses one fewer bit of entropy per entry. We also provide an iterative refinement procedure for the approximation solution that, in our experiments, yields near-optimal results while retaining a substantial speed advantage over our solver for the optimal solution.

量化压缩熵约束动态规划

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。