用随机哈达玛变换实现可证明的量化,速度更快且误差更小。
Provable Quantization with Randomized Hadamard Transform
- 在哈达玛变换后加随机偏移,提升量化精度。
- 误差随比特数增加呈4^{-b}衰减,接近理想随机旋转效果。
- 适合对效率和理论保证都有要求的压缩场景。
通过随机投影结合标量量化是机器学习中的基础技术,应用于相似性搜索、联邦学习和键值缓存压缩等任务。虽然稠密随机旋转具有良好的理论保证,但计算复杂度为Θ(d²);而随机哈达玛变换(HD)将成本降至O(d log d),其离散结构使分析困难,压缩保证较弱或仅靠经验验证。本文研究一种变体:单次随机哈达玛变换下的加抖动量化。具体地,在输入向量上应用HD后,减去一个随机标量偏移再进行量化,以极低开销引入额外随机性。我们证明该方法无偏,并提供均方误差界,渐近上与真正随机旋转矩阵相当。特别地,证明了抖动版TurboQuant在每坐标b比特下,均方误差为(π√3/2 + o(1))·4^{-b},其中o(1)项在所有单位向量和维度下一致趋于零,当量化级数增大时。
原文摘要 · Abstract (English)
Vector quantization via random projection followed by scalar quantization is a fundamental primitive in machine learning, with applications ranging from similarity search to federated learning and KV cache compression. While dense random rotations yield clean theoretical guarantees, they require $Θ(d^2)$ time. The randomized Hadamard transform $HD$ reduces this cost to $O(d \log d)$, but its discrete structure complicates analysis and leads to weaker or purely empirical compression guarantees. In this work, we study a variant of this approach: dithered quantization with a single randomized Hadamard transform. Specifically, the quantizer applies $HD$ to the input vector and subtracts a random scalar offset before quantizing, injecting additional randomness at negligible cost. We prove that this approach is unbiased and provides mean squared error bounds that asymptotically match those achievable with truly random rotation matrices. In particular, we prove that a dithered version of TurboQuant achieves mean squared error $\bigl(π\sqrt{3}/2 + o(1)\bigr) \cdot 4^{-b}$ at $b$ bits per coordinate, where the $o(1)$ term vanishes uniformly over all unit vectors and all dimensions as the number of quantization levels grows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。