基于费舍尔信息的自适应量化，提升大模型在普通显卡上的推理性能。
FAMPWQ: Fisher Information-based Adaptive Mixed Precision Weight Quantization for Effective LLM Inference
- 用费舍尔信息衡量各层对量化敏感度，实现分层自适应精度分配。
- 在7个模型5个基准上，最高提升6.87%准确率，困惑度降低3.39。
- 适合需要高效部署大模型的开发者，尤其关注推理性能优化者。
近年来，大语言模型（LLMs）在多个领域取得显著进展，但其高昂的资源需求阻碍了在资源受限设备上的部署。尽管模型量化是一种有效方法，但传统量化常因统一比特位或简单启发式敏感性评估导致性能严重下降。本文提出一种基于费舍尔信息的自适应混合精度权重量化方法（FAMPWQ），实现面向通用GPU的有效大模型推理。首先，构建包含新费舍尔信息度量的系统模型，用于衡量各层对量化的敏感度；其次，提出基于强化学习的比特位分配器，根据费舍尔信息敏感度生成自适应比特分配策略。在7个模型和5个基准上的大量实验表明，FAMPWQ在困惑度（PPL）上相比7个基线方法最高降低3.39，在准确率上最高提升6.87%，在大模型作为裁判的对比中最高获得76%胜率。
原文摘要 · Abstract (English)
Recent years have witnessed remarkable achievements of Large Language Models (LLMs) in multiple domains, while the excessive resource requirements of LLMs hinder the deployment on resource-constrained devices. Although model quantization stands out as an effective approach, conventional quantization approaches typically incur severe performance degradation due to uniform bit-width or simple heuristic sensitivity evaluation. In this paper, we propose a novel Fisher information-based Adaptive Mixed Precision Weight Quantization approach, i.e., FAMPWQ, which performs layer-adaptive weight quantization for effective LLM inference on commodity GPUs. First, we propose a system model with a novel Fisher information metric to measure the layer-wise sensitivity to quantization. Second, we propose a reinforcement learning-based bit-width allocator in FAMPWQ, which generates an adaptive bit-width allocation strategy based on the Fisher information sensitivity metric. Extensive experiments on 7 models and 5 benchmarks demonstrate that FAMPWQ significantly outperforms 7 baseline approaches in terms of PPL (up to 3.39 smaller), accuracy (up to 6.87% higher), and LLM-as-a-judge comparison (up to 76% win rate).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。