通过识别任务关键权重,实现高效低比特模型压缩。
Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression
- 基于任务相关权重电路选择性保留16位精度,其余量化
- 3.1比特下恢复Llama-3 96%性能,比SPQR高5.25%
- 无需特定任务数据也能识别重要权重,适用性广
后训练量化(PTQ)可在不重新训练的情况下将模型权重压缩至低位宽,但低至2-3比特时性能下降显著。本文提出混合精度PTQ方法Task-Circuit Quantization(TaCQ),借鉴电路发现思想,直接识别与下游任务性能相关的权重子集(即任务电路),将其保留为16位,其余权重量化,仅增加少量内存开销。TaCQ通过对比未量化与全量化模型的权重变化,并利用梯度信息预测量化对任务性能的影响,从而精准保留关键权重。在Llama-3和Qwen2.5的问答、数学推理与文本转SQL任务上,相比基线方法,TaCQ在相同校准数据和更低权重预算下表现更优:3.1比特时恢复96%的16位MMLU性能,较SPQR提升5.25%;2比特下平均优于最强基线SliM-LLM 14.74%;即使不依赖特定任务数据,仍实现7.20%性能增益,证明其具备通用重要权重识别能力。
原文摘要 · Abstract (English)
Post-training quantization (PTQ) reduces a model's memory footprint by mapping full precision weights into low bit weights without costly retraining, but can degrade its downstream performance especially in low 2- to 3-bit settings. We develop a new mixed-precision PTQ approach, Task-Circuit Quantization (TaCQ), that draws parallels to automated circuit discovery, directly conditioning the quantization process on specific weight circuits -- which we define as sets of weights associated with downstream task performance. These weights are kept as 16-bit weights, while others are quantized, maintaining performance while only adding a marginal memory cost. Specifically, TaCQ contrasts unquantized model weights with a uniformly-quantized model to estimate the expected change in weights due to quantization and uses gradient information to predict the resulting impact on task performance, allowing us to preserve task-specific weights. We compare TaCQ-based quantization to existing mixed-precision quantization methods when conditioning both on general-purpose and task-specific data. Across QA, math reasoning, and text-to-SQL tasks for both Llama-3 and Qwen2.5, we find that TaCQ outperforms baselines using the same calibration data and a lower weight budget, achieving major improvements in the 2 and 3-bit regime. With only 3.1 bits we are able to recover 96% of Llama-3-8B-Instruct's unquantized 16-bit MMLU performance, obtaining a 5.25% absolute improvement over SPQR. We also observe consistently large gains over existing methods in the 2-bit regime, with an average gain of 14.74% over the strongest baseline, SliM-LLM. Moreover, we observe a 7.20% gain without conditioning on specific tasks, showing TaCQ's ability to identify important weights is not limited to task-conditioned settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。