研究无校准信息下的高率量化矩阵乘法，揭示了量化率与失真间的理论极限。
High-Rate Quantized Matrix Multiplication I
- 在无先验统计信息下，研究双端量化矩阵乘法的高率理论边界
- 发现传统INT和浮点量化方案性能接近理论极限，且提出精确近似公式
- 适用于大模型部署中低精度计算优化，尤其适合缺乏校准数据场景
本文研究量化矩阵乘法（MatMul）问题，该问题对大规模语言模型（LLMs）的高效部署至关重要。我们考虑通用MatMul设置，即两个矩阵均需量化（权重+激活量化），且不依赖于因子的先验统计信息（无校准）。我们回顾了量化率与失真之间的基本信息论权衡（高率理论），并对比了主流量化方案（absmax INT与浮点（FP））的性能，同时为这些方案推导出准确的启发式近似。本文第二部分研究仅权重量化的情形，其中编码器可获取激活矩阵的二阶统计信息。
原文摘要 · Abstract (English)
This paper investigates the problem of quantized matrix multiplication (MatMul), which has become crucial for the efficient deployment of large language models (LLMs). We consider a Generic MatMul setting, where both matrices must be quantized (weight+activation quantization) without specific apriori (calibration) statistical information about the factors. We review the fundamental information-theoretic tradeoff between quantization rate and distortion (high-rate theory), and contrast those with the performance of popular quantization schemes (absmax INT and floating-point (FP)), for which we also derive accurate heuristic approximations. Part II of this paper studies the weight-only quantization setup where second-order statistics of the activation matrices are available at the encoder.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。