arXiv:2604.18555cs.LGcs.AI2026-04被引 6

揭示了新量化方法与早期工作关系,证明其存在更优替代方案。

A Note on TurboQuant and the Earlier DRIVE/EDEN Line of Work

  • TurboQuant是EDEN的特例,固定缩放参数导致性能下降
  • 优化缩放因子后,EDEN在多种设置下均优于TurboQuant
  • 实验表明2比特EDEN可超越3比特TurboQuant_prod

本文厘清了近期的TurboQuant工作与早期DRIVE(NeurIPS 2021)和EDEN(ICML 2022)方法的关系。DRIVE为1比特量化器,EDEN将其扩展至任意b>0比特每坐标,统称为EDEN。TurboQuant_mse是通过将EDEN的标量缩放参数固定为S=1得到的特例。EDEN支持有偏和无偏量化,各自最优的S不同;而固定S=1通常次优,尽管有偏EDEN的最优S随维度增大趋近于1,因此TurboQuant_mse在高维下接近EDEN表现。TurboQuant_prod则结合$(b-1)$比特有偏EDEN与1比特无偏残差量化,存在三方面缺陷:(1) $(b-1)$比特步骤使用次优的S=1;(2) 1比特无偏残差量化误差高于无偏1比特EDEN;(3) 有偏$(b-1)$比特加1比特无偏残差的链式结构不如直接对输入无偏量化为b比特EDEN。两篇工作共享分析工具:均利用随机旋转与移位Beta分布的关联、采用Lloyd-Max算法,并指出随机哈达玛变换可替代均匀随机旋转。实验验证:优化缩放因子的有偏EDEN优于TurboQuant_mse,而无偏EDEN显著优于TurboQuant_prod,常高出超过1比特(如2比特EDEN胜过3比特TurboQuant_prod)。我们复现了TurboQuant中所有精度实验,结果均显示EDEN在所有测试配置下表现更优。

原文摘要 · Abstract (English)

This note clarifies the relationship between the recent TurboQuant work and the earlier DRIVE (NeurIPS 2021) and EDEN (ICML 2022) schemes. DRIVE is a 1-bit quantizer that EDEN extended to any $b>0$ bits per coordinate; we refer to them collectively as EDEN. First, TurboQuant$_{\text{mse}}$ is a special case of EDEN obtained by fixing EDEN's scalar scale parameter to $S=1$. EDEN supports both biased and unbiased quantization, each optimized by a different $S$ (chosen via methods described in the EDEN works). The fixed choice $S=1$ used by TurboQuant is generally suboptimal, although the optimal $S$ for biased EDEN converges to $1$ as the dimension grows; accordingly TurboQuant$_{\text{mse}}$ approaches EDEN's behavior for large $d$. Second, TurboQuant$_{\text{prod}}$ combines a biased $(b-1)$-bit EDEN step with an unbiased 1-bit QJL quantization of the residual. It is suboptimal in three ways: (1) its $(b-1)$-bit step uses the suboptimal $S=1$; (2) its 1-bit unbiased residual quantization has worse MSE than (unbiased) 1-bit EDEN; (3) chaining a biased $(b-1)$-bit step with a 1-bit unbiased residual step is inferior to unbiasedly quantizing the input directly with $b$-bit EDEN. Third, some of the analysis in the TurboQuant work mirrors that of the EDEN works: both exploit the connection between random rotations and the shifted Beta distribution, use the Lloyd-Max algorithm, and note that Randomized Hadamard Transforms can replace uniform random rotations. Experiments support these claims: biased EDEN (with optimized $S$) is more accurate than TurboQuant$_{\text{mse}}$, and unbiased EDEN is markedly more accurate than TurboQuant$_{\text{prod}}$, often by more than a bit (e.g., 2-bit EDEN beats 3-bit TurboQuant$_{\text{prod}}$). We also repeat all accuracy experiments from the TurboQuant paper, showing that EDEN outperforms it in every setup we have tried.

量化深度学习压缩机器学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。