4-bit量化时保持模型安全对齐,避免精度高却不安全的隐患。
Safety-Preserving PTQ via Contrastive Alignment Loss
- 引入对比对齐损失,同时优化量化模型的分布一致性和安全行为。
- 在4比特量化下,多个大模型安全对齐优于现有最先进方法。
- 无需额外安全数据,仅用常规校准数据,计算开销几乎不变。
后训练量化(PTQ)已成为高效部署大语言模型的主流方式,但其优化目标仍不完整。现有方法仅最小化重建误差(如MSE或KL散度),未考虑通过安全微调获得的行为对齐——这一关键特性。我们发现,这种目标不匹配会导致系统性漏洞:模型可维持低困惑度,但安全对齐显著退化,说明困惑度本身不足以衡量部署就绪状态。为此,我们提出对比对齐量化(CAQ),通过引入对比对齐损失(CAL)扩展了PTQ的目标设计空间。CAL采用原则化的推拉机制,联合优化分布保真度与行为对齐:使量化模型趋向安全指令微调版本,同时远离未对齐的预训练参考模型。CAQ无需专用安全数据集,仅依赖标准校准数据,且对现有基于变换的PTQ流水线引入可忽略的计算开销。实验表明,CAQ在多种模型族(包括LLaMA、Qwen和Mistral)上实现了鲁棒的4比特(W4A4)量化,在现有最佳PTQ方法失效的场景下仍保持优越的安全对齐能力,且不牺牲通用性能。匿名代码见附录。
原文摘要 · Abstract (English)
Post-Training Quantization (PTQ) has become the de-facto standard for efficient LLM deployment, yet its optimization objective remains fundamentally incomplete. Standard PTQ methods minimize reconstruction error (e.g., MSE or KL divergence) without accounting for behavioral alignment--a critical property instilled through safety fine-tuning. We demonstrate that this objective mismatch introduces a systematic vulnerability: models can maintain low perplexity yet exhibit significant degradation in safety alignment, revealing that perplexity alone is an insufficient and often misleading proxy for deployment readiness. To address this, we propose Contrastive Alignment Quantization (CAQ), which extends the PTQ objective design space by integrating a Contrastive Alignment Loss (CAL). CAL introduces a principled push-pull mechanism that jointly optimizes distributional fidelity and behavioral alignment: it steers the quantized model toward its safe, instruction-tuned counterpart while diverging from the unaligned, pre-trained reference. CAQ requires no specialized safety datasets, relying solely on standard calibration data, and introduces negligible computational overhead over existing transformation-based PTQ pipelines. We show that CAQ enables robust 4-bit (W4A4) quantization across diverse model families--including LLaMA, Qwen, and Mistral--achieving superior safety alignment where state-of-the-art PTQ methods fail, without sacrificing general capabilities. Anonymized code is available in the supplementary material.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。