量化让代码生成大模型更抗干扰,反而更可靠。
Smaller = Weaker? Benchmarking Robustness of Quantized LLMs in Code Generation
- 对比4种模型、350M到33B参数规模,系统测试量化对代码生成鲁棒性的影响。
- 51.59%的对抗攻击实验中,量化模型比原始模型更抗干扰,噪声实验也显示更强稳定性。
- 挑战“量化即退化”认知,适合部署优化与模型可靠性研究者参考。
量化已成为压缩大语言模型(LLMs)的主流方法,可在不修改架构的前提下降低内存占用并加速推理。现有研究多关注量化模型与原始模型在性能上的差异,但对其鲁棒性影响仍缺乏系统探索。本文首次系统考察量化对代码生成任务中LLM鲁棒性的影响。我们在四个主流模型家族(LLaMA、DeepSeek、CodeGen、StarCoder)上,覆盖350M至33B参数规模,从双重角度评估鲁棒性:对输入提示的对抗攻击,以及对模型权重的噪声扰动。结果表明,量化后的模型通常表现出更强的鲁棒性:在51.59%的对抗攻击实验中,量化模型表现优于全精度版本;在噪声扰动实验中,量化模型也能承受更高程度的权重偏差。这说明量化不仅降低计算开销,还可能提升代码生成任务中的可靠性,为构建更高效、更鲁棒的LLM部署策略提供重要依据。
原文摘要 · Abstract (English)
Quantization has emerged as a mainstream method for compressing Large Language Models (LLMs), reducing memory requirements and accelerating inference without architectural modifications. While existing research primarily focuses on evaluating the effectiveness of quantized LLMs compared to their original counterparts, the impact on robustness remains largely unexplored.In this paper, we present the first systematic investigation of how quantization affects the robustness of LLMs in code generation tasks. Through extensive experiments across four prominent LLM families (LLaMA, DeepSeek, CodeGen, and StarCoder) with parameter scales ranging from 350M to 33B, we evaluate robustness from dual perspectives: adversarial attacks on input prompts and noise perturbations on model architecture. Our findings challenge conventional wisdom by demonstrating that quantized LLMs often exhibit superior robustness compared to their full-precision counterparts, with 51.59% versus 42.86% of our adversarial experiments showing better resilience in quantized LLMs. Similarly, our noise perturbation experiments also confirm that LLMs after quantitation generally withstand higher levels of weight disturbances. These results suggest that quantization not only reduces computational requirements but can actually enhance LLMs' reliability in code generation tasks, providing valuable insights for developing more robust and efficient LLM deployment strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。