低比特量化会严重损害大模型数学推理能力,该研究定位问题源头并精准修复。
Quantization Meets Reasoning: Exploring and Mitigating Degradation of Low-Bit LLMs in Mathematical Reasoning
- 通过步骤级归因定位量化导致错误的最早环节。
- 仅用332个示例和数分钟算力,4比特模型推理准确率恢复至原始水平。
- 方法不依赖特定架构或量化器,适合实际部署场景。
低比特后训练量化(PTQ)是资源受限环境下部署具备推理能力的大语言模型的实用方案,但可能显著削弱数学推理性能(在更难设置下下降高达69.81%)。本文针对两个关键部署问题进行过程级分析:错误何时首次出现?如何在保持低比特的前提下缓解?在多种量化方法(AWQ、GPTQ、SmoothQuant)、开源模型(Qwen、LLaMA;0.5–7B)和数学推理基准(GSM8K、MATH、AIME)上,采用格式对齐的思维链与步骤级归因,发现两条稳健规律:(i) 量化使方法与执行错误显著增加,远超高层次概念错误;(ii) 失败早发于解题首步,错误逐级传导至最终答案。据此提出“定位→修复”干预原则:在最早故障点精准恢复局部词元置信度。我们实现为轻量级测量→定位→恢复循环,直接作用于量化模型:检测首个故障步骤,构建‘银弹’数据集,并进行小规模监督/偏好微调。在实验条件下,仅需332个精选样本及单卡3–5分钟计算,即可使4比特权重模型的数学推理性能恢复至全精度基线,同时保持量化效率。该框架在评估范围内对量化器与架构均具兼容性,将低比特退化从全局精度问题转化为可复现的局部过程干预。
原文摘要 · Abstract (English)
Low-bit post-training quantization (PTQ) is a practical route to deploy reasoning-capable LLMs under tight memory and latency budgets, yet it can markedly impair mathematical reasoning (drops up to 69.81% in our harder settings). We address two deployment-critical questions with process-level precision: Where along a step-structured solution does degradation first arise? How to mitigate it while staying in the low-bit regime? Across widely used PTQ methods (AWQ, GPTQ, SmoothQuant), open-source model families (Qwen, LLaMA; 0.5--7B), and math reasoning benchmarks (GSM8K, MATH, AIME), we perform format-aligned chain-of-thought with step-aligned attribution and uncover two robust regularities: (i) PTQ disproportionately elevates method and execution errors relative to high-level conceptual mistakes; and (ii) failures emerge early, with the first vulnerable step flipping and cascading to the final answer. These regularities suggest a general intervention principle: restore local token-level margins exactly at the earliest failure frontier. We instantiate this principle as a lightweight measure$\rightarrow$locate$\rightarrow$restore loop that operates directly on the quantized model: detect the first faulty step, construct our "Silver Bullet" datasets, and apply small-scale supervised/preference tuning. In our settings, as few as 332 curated examples and 3--5 minutes of compute on a single GPU recover 4-bit weight math reasoning toward the full-precision baseline while preserving PTQ efficiency. Our framework is quantizer- and architecture-agnostic within the evaluated regimes, and turns low-bit degradation from a global accuracy problem into a local, reproducible process intervention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。