4-bit量化让大模型在重复更新时记忆衰退更严重，影响长期上下文任务。
Compress and Forget: bitsandbytes Quantization Amplifies Proactive Interference in LLMs
- 用bitsandbytes对模型进行4位量化，放大了前向干扰现象
- Qwen模型在高干扰下准确率从81.0%降至68.3%，统计显著（p≤2.6e-6）
- 适合关注长上下文、频繁更新任务的开发者注意，即使整体性能没变
前向干扰（PI）是大语言模型中一种已知的失效模式，即反复覆盖同一值后，其检索准确率随历史覆盖次数增加而下降，类似人类工作记忆中的经典现象。后训练量化（PTQ）现已成为开源模型部署的标准路径，但其对这一失效模式的影响尚未被检验。我们在三种架构不同的指令微调模型（Qwen2.5-7B-Instruct、Mistral-7B-Instruct-v0.3、Phi-3.5-mini-instruct）上评估了三种精度（FP16、INT8、INT4/NF4，通过bitsandbytes实现），固定检索任务。结果显示，所有模型在高干扰条件下，INT4量化均显著降低准确率（如Qwen从81.0%降至68.3%），经配对McNemar检验（p ≤ 2.6 × 10⁻⁶）和混合效应回归验证。尽管如此，常被认为安全的INT8在其中两模型中仍带来轻微但真实的影响。该效应仅出现在语义相似（词性类型）的干扰项中，并在数值控制条件下符号反转；机制上与INT4下同键侵入错误上升相关（从21.5%增至24.6%，p = 4.8 × 10⁻⁷）。消融实验表明，该效应源于量化后的Transformer主干网络，而非输出投影层。结果表明，bitsandbytes 4-bit量化可能为依赖长期可更新语义密集上下文的应用带来额外代价，即使整体基准准确率看似未受影响。代码与分词器验证的词汇构建方法已公开于https://github.com/ShayanShahrabi/compress-and-forget。
原文摘要 · Abstract (English)
Proactive interference (PI) is a documented failure mode in large language models in which retrieval of a repeatedly overwritten value degrades as prior overwrites accumulate, mirroring a classical phenomenon in human working memory. Post-training quantization (PTQ) is now the default deployment path for open-weight models, yet its effect on this failure mode has not been tested. We evaluate three precision levels (FP16, INT8, INT4/NF4, via bitsandbytes) across three architecturally distinct instruction-tuned models (Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, Phi-3.5-mini-instruct), holding the retrieval task fixed. INT4 quantization significantly reduces accuracy under high interference in every model (e.g., from 81.0% to 68.3% for Qwen), confirmed by paired McNemar's tests ($p \le 2.6 \times 10^{-6}$) and a mixed-effects regression spanning all interference levels; INT8, often assumed safe, also carries a smaller but real penalty in two of three models. The effect is specific to semantically similar (word-type) distractors and reverses sign under a numeric control condition, and is mechanistically linked to a rise in same-key intrusion errors under INT4 (from 21.5% to 24.6% of trials, $p = 4.8 \times 10^{-7}$). A follow-up ablation shows the effect originates in the quantized transformer backbone rather than the output projection layer. These results suggest that bitsandbytes 4-bit quantization can impose an additional cost on applications relying on long, updatable, semantically dense contexts, even when aggregate benchmark accuracy appears largely unaffected. We release our code and tokenizer-verified vocabulary construction method at https://github.com/ShayanShahrabi/compress-and-forget
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。