提出可学习的4比特表示，让大模型压缩后更准更快。
any4: Learned 4-bit Numeric Representation for LLMs
- 通过可学习的4比特表示，无需预处理权重和激活值。
- 在多个模型上表现优于int4/fp4/nf4，精度更高。
- 仅需少量样本校准，适合部署于资源受限场景。
我们提出 any4，一种用于大语言模型（LLMs）的可学习4比特权重量化方法，支持任意数值表示，无需对权重或激活值进行预处理。any4 在多种模型规模、生成任务及模型家族（Llama 2、Llama 3、Mistral、Mixtral）上均优于其他4比特表示形式（如 int4、fp4、nf4），在精度上表现更优。尽管无需预处理，其性能仍可媲美需要预处理的技术（如 AWQ、GPTQ）。我们还探索了 any3 和 any2，证明其在更低比特下依然具备竞争力。此外，我们发现仅需一个精选的多样化样本即可完成校准，远少于传统方法所需的数百个样本。我们开源了 tinygemm，一个针对 LLM 优化的 GPU 矩阵乘法库，采用高效的查表策略实现 any4 及其他量化方法。代码已开放：https://github.com/facebookresearch/any4。
原文摘要 · Abstract (English)
We present any4, a learned 4-bit weight quantization solution for large language models (LLMs) providing arbitrary numeric representations without requiring pre-processing of weights or activations. any4 yields higher accuracy compared to other related 4-bit numeric representation types: int4, fp4 and nf4, as evaluated on a range of model sizes, generations and families (Llama 2, Llama 3, Mistral and Mixtral). While any4 does not require preprocessing of weights or activations, it is also competitive with orthogonal techniques that require such preprocessing (e.g., AWQ and GPTQ). We also experiment with any3 and any2 and show competitiveness at lower bits. Additionally, we show that we can calibrate using a single curated diverse sample rather than hundreds of samples from a dataset as done in most quantization approaches. We also open source tinygemm, a latency optimized GPU matrix multiplication library for LLMs, that implements any4 using a GPU-efficient lookup table strategy along with other common quantization methods. We open source our code at https://github.com/facebookresearch/any4 .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。