用多语言重要性矩阵量化大模型，不影响多语种性能。
English K_Quantization of LLMs Does Not Disproportionately Diminish Multilingual Performance
- 用英、挪、马拉雅拉姆三语构建重要性矩阵进行量化。
- 在混杂评估集上，英语与挪威语任务表现无显著差异。
- 适合关注多语言模型部署的开发者和研究者。
对于本地部署的大语言模型，GGUF格式和k量化是关键工具，可在保持原模型性能的同时，将模型压缩至消费级硬件可运行的大小。量化通过减少各权重的比特数实现，依据其在推理中的重要性，而该重要性由一个代表典型使用场景的小型文本文档决定。目前绝大多数量化模型使用的文档均为英文，因此存在疑问：量化是否会导致多语言性能下降？本文通过在英语、挪威语和马拉雅拉姆语三种语言的重要性矩阵上对Llama3.3 70B进行量化，并在MixEval数据集上评估其在英语和挪威语上的表现，结果显示所有实验均未出现显著差异，表明当前量化方法不会不成比例地损害多语言性能。
原文摘要 · Abstract (English)
For consumer usage of locally deployed LLMs, the GGUF format and k\_quantization are invaluable tools for maintaining the performance of the original model while reducing it to sizes deployable with consumer-grade hardware. The number of bits dedicated to each weight from the original model is reduced based on how important they are thought to be during model inference. This importance is arrived at through the application of an 'importance matrix'-a relatively small text document meant to be representative of the LLM's standard use-cases. In the vast majority of quants available online, this document is primarily written in English. It was therefore an open question whether performance on English language tasks was preserved through the sacrifice of multilingual performance and whether it can be preserved with alternate importance matrices. This article investigates these hypotheses by quantizing Llama3.3 70B on importance matrices written in three languages (English, Norwegian, and Malayalam) and evaluating them on the MixEval dataset in both English and Norwegian. All experiments related to yielded non-significant results indicating that current quantization practices do not disproportionately harm multilingual performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。