arXiv:2607.14181cs.SEcs.LG2026-07

实测六种量化方法对代码生成质量影响,选错会大幅降低正确率。

Quantize with Confidence? An Empirical Study of Quantization for Code Generation

论文配图:Quantize with Confidence? An Empirical Study of Quantization for Code Generation
图 1 · 摘自论文原文
  • 对比六种量化方法在代码模型上的表现
  • AQLM几乎不降质,QuIP#在复杂提示下错误率飙升
  • 关注代码可维护性、安全性及对提示复杂度的鲁棒性

随着Ollama等本地推理框架普及,开发者越来越多地在笔记本等资源受限设备上运行大模型。后训练量化对减少内存占用至关重要,但其对生成代码的影响仍不明确。本文针对Qwen2.5-Coder和CodeLlama两个主流代码模型家族,使用多语言McEval和CoderEval基准(涵盖Python与Java),评估六种先进量化方法(GPTQ、AWQ、QuIP#、AQLM、BitsAndBytes、GGUF)在功能正确性(pass@1)、可维护性、可靠性、安全性及结构复杂度上的表现。我们还引入新分析:基于香农熵和词元长度,考察提示复杂度下的鲁棒性。结果表明,不同量化方法对正确性和代码质量影响显著差异。其中,AQLM始终匹配或优于全精度基线;QuIP#在复杂提示下出现最大正确率下降。安全属性在各模型、基准与语言间保持稳定,但对提示复杂度的鲁棒性因方法而异。研究为资源受限硬件部署提供量化策略选择依据,并强调应超越功能正确性全面评估量化模型。

原文摘要 · Abstract (English)

The growing adoption of local inference frameworks such as Ollama has made it increasingly common for developers to run large code models on laptops and other resource-constrained hardware. In these settings, post-training quantization is essential for reducing memory footprint and enabling practical deployment, yet its impact on generated code remains insufficiently understood. We empirically evaluate six state-of-the-art quantization methods (GPTQ, AWQ, QuIP#, AQLM, BitsAndBytes, and GGUF) on two representative large code model families, Qwen2.5-Coder and CodeLlama, using the multilingual McEval and CoderEval benchmarks for Python and Java. We assess functional correctness (pass@1) together with maintainability, reliability, security, and structural complexity. We also introduce a novel analysis of robustness under varying prompt complexity, characterized by Shannon entropy and token length. Our results show that quantization techniques differ meaningfully in their impact on correctness and code quality. AQLM consistently matches or exceeds the full-precision baseline, whereas QuIP# exhibits the largest correctness degradation, particularly on complex prompts. Security attributes remain stable across models, benchmarks, and programming languages, while robustness to prompt complexity varies across techniques. These findings provide practical guidance for selecting quantization strategies for deploying large code models on resource-constrained hardware and highlight the importance of evaluating quantized models beyond functional correctness.

代码生成量化模型压缩推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。