量化让推理模型更啰嗦,反而增加实际计算成本。
Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models

- 量化后模型虽答对但推理步骤变多,产生隐性开销。
- INT4/INT3量化使推理词元数增加,抵消速度优势。
- 适合关注推理效率与真实部署性能的研究者。
量化被广泛用于降低大语言模型的推理成本,但其对推理模型的影响无法仅通过最终答案准确率或每词元延迟完全体现。我们发现,低比特后训练量化会引入隐藏的运行时计算开销:量化后的推理模型在仍能正确作答的情况下,往往生成更长的思维链。在数学推理、代码生成、科学问答及代理工具使用等多个基准测试中,INT4/INT3量化可保持准确率,但显著增加推理词元使用量,抵消了预期的每词元加速。为此,我们提出思维链词元膨胀比(CoT Token Inflation Ratio),用于衡量量化与全精度模型在所有评估基准上推理长度的平均差异。进一步分析显示,词元膨胀伴随推理轨迹的行为变化,包括更多中间步骤和更高的语义重复度。这些变化带来可测量的端到端服务延迟惩罚。最后,我们评估了缓解策略,发现提示工程和解码时采样效果不稳定,而量化感知训练在减少准确率下降与词元膨胀方面更具潜力。结果表明,评估量化推理模型时应同时报告准确率与推理词元用量。
原文摘要 · Abstract (English)
Quantization is widely used to reduce the inference cost of large language models, but its effect on reasoning models is not fully captured by final-answer accuracy or per-token latency. We show that low-bit post-training quantization can introduce a hidden test-time compute cost: quantized reasoning models often generate longer chains of thought even when they still answer correctly. Across mathematical reasoning, code generation, scientific question answering, and agentic tool-use benchmarks, we find that INT4/INT3 quantization can preserve accuracy but increase reasoning-token usage, offsetting the expected per-token speedup. To measure this effect, we introduce the CoT Token Inflation Ratio, which compares reasoning length between quantized and full-precision models averaged across all evaluation benchmarks. We further show that token inflation is accompanied by behavioral changes in the reasoning trace, including more intermediate steps and greater semantic repetition. These changes translate into measurable end-to-end real-world serving penalties. Finally, we evaluate mitigation strategies and find that prompting and decoding-time sampling offer inconsistent accuracy-length trade-offs, while quantization-aware training shows more promise in reducing both accuracy degradation and token inflation. Our results suggest that reasoning-token usage should be reported alongside accuracy when evaluating quantized reasoning models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。