提出WaterSIC算法，实现线性层量化在信息论下的近乎最优压缩。
WaterSIC: Information-Theoretically (Near) Optimal Linear Layer Quantization
- 按输入特征列分配不同量化精度，模仿信息论中的水填法。
- 在1到4比特下，对Llama和Qwen模型均达新最佳性能。
- 理论证明该方法距信息论极限仅差0.255比特，普适于所有输入协方差。
本文研究将稠密线性层转换为低精度的问题，从信息论角度分析压缩长度与输出偏差之间的权衡。研究表明，流行的GPTQ算法可能与信息论极限存在任意大的差距。为此，提出一种新算法WaterSIC，其在所有可能的输入激活协方差矩阵下，始终与信息论极限保持不超过0.255比特的速率差距。WaterSIC的核心创新在于对权重矩阵的不同列（输入特征）分配不同的量化率，模仿经典的信息论解法——水填法。将WaterSIC应用于Llama和Qwen系列大模型，在1至4比特的各类量化率下均取得当前最优性能。代码已开源：https://github.com/egorlifar/watersic。
原文摘要 · Abstract (English)
This paper considers the problem of converting a given dense linear layer to low precision. The tradeoff between compressed length and output discrepancy is analyzed information theoretically (IT). It is shown that a popular GPTQ algorithm may have an arbitrarily large gap to the IT limit. To alleviate this problem, a novel algorithm, termed ``WaterSIC'', is proposed and is shown to be within a rate gap of 0.255 bits to the IT limit, uniformly over all possible covariance matrices of input activations. The key innovation of WaterSIC's is to allocate different quantization rates to different columns (in-features) of the weight matrix, mimicking the classical IT solution known as "waterfilling". Applying WaterSIC to the Llama and Qwen family of LLMs establishes new state-of-the-art performance for all quantization rates from 1 to 4 bits. Our code is available at https://github.com/egorlifar/watersic.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。