arXiv:2605.06014cs.LGcs.AI2026-05被引 4

用两次随机哈达玛变换逼近最优旋转,提升量化效率与稳定性

Quantizing With Randomized Hadamard Transforms: Efficient Heuristic Now Proven

  • 通过双重随机哈达玛变换实现接近理想旋转的分布特性
  • 两次变换可使坐标分布误差小于 $O(d^{-1/2})$,媲美均匀随机旋转
  • 动态检测输入特征,按需调整变换次数以优化实际性能

均匀随机旋转(URRs)是现代量化方法中常见的预处理步骤,广泛应用于梯度压缩、推理加速、键值缓存压缩、模型权重量化及向量数据库中的近似最近邻搜索。实践中常采用随机哈达玛变换(RHTs)替代,因其保持正交性且计算高效。然而,对最坏情况输入的性能仍存疑。我们证明:对任意 $d$ 维输入向量,经过两次 RHT 后,归一化后向量任一固定坐标的边际分布,在 Kolmogorov 距离和 1- Wasserstein 距离下均与标准高斯分布相差 $O(d^{-1/2})$。将该界应用于 DRIVE 与 QUIC-FL 等压缩方案,表明两次 RHT 可实现渐近等价于 URR 的性能。但对向量量化(VQ),仅边际分布收敛不足;我们证明三重 RHT 可带来坐标协方差衰减,使得针对 URR 设计的任意有界多维码本,其期望误差在三重 RHT 下仅增加一个随维度趋零的项。此外,由于实际输入极少为对抗性,我们提出线性时间 ${O}(d)$ 的输入矩检测机制,动态调整运行时 RHT 次数以提升性能。

原文摘要 · Abstract (English)

Uniform random rotations (URRs) are a common preprocessing step in modern quantization approaches used for gradient compression, inference acceleration, KV-cache compression, model weight quantization, and approximate nearest-neighbor search in vector databases. In practice, URRs are often replaced by randomized Hadamard transforms (RHTs), which preserve orthogonality while admitting fast implementations. The remaining issue is the performance for worst-case inputs. With a URR, each coordinate is individually distributed as a shifted beta distribution, which converges to a Gaussian distribution in high dimensions. Generally, one RHT is not suitable in the worst case, as individual coordinates can be far from these distributions. We show that after composing two RHTs on any $d$-sized input vector, the marginal distribution of every fixed coordinate of the normalized rotated vector is within $O(d^{-1/2})$ of a standard Gaussian both in Kolmogorov distance and in $1$-Wasserstein distance. We then plug these bounds into the analyses of modern compression schemes, namely DRIVE and QUIC-FL, and show that two RHTs achieve performance that asymptotically matches URRs. However, we show that two RHTs may not be sufficient for Vector Quantization (VQ), which often requires weak correlation across fixed-size blocks of coordinates (as opposed to only marginal distribution convergence for single coordinates). We prove that a composition of three RHTs leads to decaying coordinate covariance. This ensures that any fixed, bounded, multi-dimensional VQ codebook optimized for URRs has the same expected error when using three RHTs, up to an additive term that vanishes with the dimension. Finally, because practical inputs are rarely adversarial, we propose a linear-time ${O}(d)$ check on the input's moments to dynamically adapt the number of RHTs used at runtime to improve performance.

量化哈达玛变换随机化压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。