提出矩阵乘法量化最优标量密度,揭示相关性引发的相变现象。
Optimal Scalar Quantization for Matrix Multiplication: Closed-Form Density and Phase Transition
- 基于高分辨率分析推导量化误差的精确渐近展开式。
- 发现相关性超过1/√3时,最优量化密度由单峰变为双峰。
- 适用于大模型激活值量化与矩阵乘法优化,理论指导实践。
我们研究矩阵元素级标量量化在乘法前的应用。给定矩阵 $A\in \mathbb{R}^{m\times k}$ 和 $B\in \mathbb{R}^{k\times n}$,独立使用 $K_X$ 与 $K_Y$ 个量化级别的标量量化器对 $A$、$B$ 的元素进行量化,得到 $\ ilde A, \ ilde B$ 并计算 $\ ilde C = \ ilde A\ ilde B$。目标是最小化矩阵乘法均方误差 $\mathbb{E}[\|AB - \ ilde A\ ilde B\|_F^2]$,在成对独立同分布内积模型下。在高分辨率极限 $K_X, K_Y \to \infty$ 时,我们推导出误差 $\mathcal{E}$ 的尖锐 $K^{-2}$ 渐近展开式,确定最优首项常数,并以条件二阶矩刻画渐近最优量化中心密度。针对相关高斯乘积对,我们获得闭式最优点密度: $$ λ^\star(u) \propto \exp\left(-\frac{u^2}{6}\right)\bigl((1-ρ^2)+ρ^2u^2\bigr)^{1/3}, \quad u=\frac{x}{σ_X}, $$ 对 $y/σ_Y$ 具有相同形式。证明存在相关性驱动的相变:当 $|ρ| \leq 1/\sqrt{3}$ 时密度为单峰,当 $|ρ| > 1/\sqrt{3}$ 时变为双峰,峰值位于 $u_{\mathrm{peak}} = \pm\sqrt{3 - 1/ρ^2}$。我们在合成实验中验证了该方法在矩阵乘法量化、最小二乘优化及大语言模型键/查询激活量化中的适用性。
原文摘要 · Abstract (English)
We study entrywise scalar quantization of two matrices prior to multiplication. Given $A\in R^{m\times k}$ and $B\in R^{k\times n}$, we quantize entries of $A$ and $B$ independently using scalar quantizers with $K_X$ and $K_Y$ levels per entry, and form $\widehat C=\widehat A\,\widehat B$. The objective is to minimize the matrix multiplication mean-squared error (MSE) $E[\|{AB-\widehat A\widehat B}\|_F^2]$ under a pair-i.i.d.\ inner-product model. In the high-resolution regime $K_X,K_Y\to\infty$, we derive a sharp $K^{-2}$ asymptotic expansion for $\mathcal{E}$, identify the exact optimal leading constants, and characterize asymptotically optimal quantization center densities in terms of conditional second moments. We then specialize to correlated Gaussian multiplicative pairs, obtaining a closed-form optimal point density \[ λ^\star(u)\ \propto\ \exp\!\left(-\frac{u^2}{6}\right)\bigl((1-ρ^2)+ρ^2u^2\bigr)^{1/3}, \qquad u=\frac{x}{σ_X}, \] with the same form for $y/σ_Y$, and prove a correlation-driven phase transition: the density is unimodal at the origin for $|ρ|\leq 1/\sqrt{3}$ and becomes bimodal for $|ρ|>1/\sqrt{3}$ with peaks at $u_{\mathrm{peak}}=\pm\sqrt{3-1/ρ^2}$. We show our method's applicability in synthetic experiments such as matrix multiplication quantization and least squares optimization, as well as quantization of large language model key and query activations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。