通过微调时的权重变化信号,实现大推理模型的高效量化。
QuantLRM: Quantization of Large Reasoning Models via Fine-Tuning Signals
- 利用微调过程中的权重更新幅度判断通道重要性
- 在强化学习微调模型上平均提升6.55%性能
- 适用于有无微调的推理模型,应用范围广
仅量化权重对压缩大语言模型至关重要。受经典重要性剪枝启发,本文研究在推理激励微调过程中,权重更新幅度是否能为大推理模型(LRMs)量化提供有效信号。我们提出‘保护两端’假设:微调中最小和最大的权重更新比中间值更重要。验证后提出QuantLRM方法,通过在权重更新上拟合简单受限二次函数以保护两端。结合通道零更新次数与平均二次值乘积,计算出更有效的通道重要性。在四个推理基准(AIME-120、FOLIO、时间序列、GPQA-Diamond)上测试多种微调模型(监督、直接偏好优化、强化学习),结果表明QuantLRM显著提升量化效果,尤其在强化学习微调模型上平均提升6.55%。同时支持非微调模型,通过伪微调获取有效信号,大幅提升适用性。
原文摘要 · Abstract (English)
Weight-only quantization is important for compressing Large Language Models (LLMs). Inspired by the spirit of classical magnitude pruning, we study whether the magnitude of weight updates during reasoning-incentivized fine-tuning can provide valuable signals for quantizing Large Reasoning Models (LRMs). We hypothesize that the smallest and largest weight updates during fine-tuning are more important than those of intermediate magnitude, a phenomenon we term "protecting both ends". Upon hypothesis validation, we introduce QuantLRM, which stands for weight quantization of LRMs via fine-tuning signals. We fit simple restricted quadratic functions on weight updates to protect both ends. By multiplying the average quadratic values with the count of zero weight updates of channels, we compute channel importance that is more effective than using activation or second-order information. We run QuantLRM to quantize various fine-tuned models (including supervised, direct preference optimization, and reinforcement learning fine-tuning) over four reasoning benchmarks (AIME-120, FOLIO, temporal sequences, and GPQA-Diamond) and empirically find that QuantLRM delivers a consistent improvement for LRMs quantization, with an average improvement of 6.55% on a reinforcement learning fine-tuned model. Also supporting non-fine-tuned LRMs, QuantLRM gathers effective signals via pseudo-fine-tuning, which greatly enhances its applicability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。